Demystify file system hierarchy for deployments
jkutej@cpan.org
16 August 2011
What to expect?
- UNIX file-systems overview - FHS
- server layouts
- ways how does the code goes from ☢ dev machine to ☑ production
- Perl code life-cycle ♲
- Perl code layers ☰
- Sys::Path + Module::Build::SysPath
- DarkPAN tools
What to expect?
- 1. OS is for you
- 2. Perl code and deployments
- 3. Related Perl modules
Why?
Why this talk?
- eliminate developer fear of the production OS
- replace if with concern about operation costs, consistency and maintainability
- show couple of tools and techniques of deployment
- to learn something new!
Why should you care?
- each project needs to be deployed.
- each successful project grows.
- growing up brings complexity.
- users today doesn't accept down-times
- it's good be feel in control of the OS, not the other way around.
- keeping to standards means no need for training of new people.
Why me?
- 3+ years of Linux system administration
- 1+ years of "corporate experience"
- 3+ years full time Perl developer
- 4+ years of packaging closed source for Debian
Why so many questions?
package Acme::KnowledgeWisdom;
use Moose;
use warnings FATAL => 'all';
has 'in_questions' => ( isa => 'Bool', is => 'ro', default => 1);
has 'has_already' => ( isa => 'Bool', is => 'ro', default => 0);
sub get {
my $kw = shift;
return $kw->ask
if $kw->in_questions;
return 42;
}
sub ask {
my $self = shift;
return 42
if $self->has_already;
my $kw = Acme::KnowledgeWisdom->new;
return $kw->get;
}
1. OS for you
Demystify file system
- cd /mnt
- mkdir bind
- mount -o bind / /mnt/bind
- mount -o bind /var /mnt/bind/var
- mount -o bind /usr /mnt/bind/usr
- ...
- rsync -avz /mnt/bind/ backup@backup-server:this-server-name/
Why not to fear OS?
- 100 ways to create & use & destroy it
- Xen, KVM, VirtualBox, VMWare, ...
- `sudo chroot /srv/chroot/machine_name/ su - you -c 'cd prog; bash'`
OS
So OS is just a bunch of files in different folders
or a collection that has no special meaning
besides making our project to run, to be live.
OS is that stuff that we need for our project.
What do we need from OS?
- Perl lib/ + bin/
- cron jobs
- daemons
- config files (self, sites-enabled, logrotate.d, …)
- lock files
- temp files
- templates
- css|js|images files
- store pre-generated data files
- .so libs
Where is Perl?
- /usr/bin/perl
- /usr/local/bin/perl
- /what-ever/bin/perl
- $HOME/local/bin/perl
- $HOME/perl5/perlbrew/perls/current/bin/perl
- C:\strawberry
- ...
Where are configurtion and data files?
- /etc/ + /usr/ + /var/ + /srv/
- /usr/local/ + /var/local/
- /opt/
- /srv/
- File::ShareDir
- File::HomeDir
- /my-project/ || /my-company/ || /local/ || /shared/ || /data/ || /.../
- ./
- "C:\Documents and Settings\All Users\Application Data"
- __DATA__
- MyApp::Config
- MyApp::ConfigData
- wherever
- don't remember, let me check
- /dev/null
- somewhere else
http://perlmonks.org/?node_id=919265
Filesystem Hierarchy Standard
OS is there for our project!
- /usr/bin/perl
- FHS (/etc/ + /usr/ + /var/ + /srv/)
- distribution packaging
2. Perl code and deployments
How to install Perl code?
- via distribution packaging
- via CPAN shell || Build install || make install -
- public from CPAN + private by hand or not at all
- from DPAN (CPAN::Mini::Inject)
- local::lib
- perlbrew
- Git::CPAN::Hook
- via checkouts - whole codebase in VC
- via project folder - rsync or NFS
- via PAR/Shipwright
Perl code layers
- unpacked raw module (lib/…/….pm)
- ↓ (Build.PL, Makefile.PL, …)
- Perl distribution (….tar.gz)
- ↓ (PAUSE, CPAN::Mini::Inject)
- CPAN || DPAN
- ↓ (patch)
- ↓ (dh-make-perl, dh-make-pm, cpan2rpm, …)
- Debian package (….deb)
- ↓ (pkg-perl.alioth.debian.org, scp, dupload, …)
- Distribution || Local reposity
- ↓ (apt-get install …)
- Production machines
Perl code life-cycle ♲
☎ customer →
☘ sales →
⚔ negotiations →
☐ specification →
✍ contract →
⚒ developer →
☢ development machine →
☀ version control →
☔ user-acceptance machine ↔
☇ quality assurance review ↔
☠ customer review →
☃ release tag →
𝄞 packaging (tarball||dist)→
★ staging machine →
☇ quality assurance review →
☑ deployment →
☇☈ quality assurance review →
☕ another job well done →
☻ happy customer →
☻☺ more customers →
$€₤¥ money && ♥♡♥♡ our job
How many server types did you count?
Depends on scale...
- minimalist
- dev (laptop) + prod = 1
- minimal
- dev (laptop) + uat&staging + 2x prod = 3
- full
- 1 x dev + code + uat + 2x staging + 2x prod = 6
- bigger
- ? x dev + code + batch + uat + 8x staging + 8x prod = ~20
- making $€₤¥
- ? x dev + code + batch + monitoring + ~4x uat + 8x staging + ~30x prod = >50
Prod?
- dev
- laptops for mobility while developing, for bigger projects
also shared dev servers needed
- code
- code repository, DPAN, Pod, wiki?, Debian repo, batch results
- batch
- auto packaging (.tar.gz, .deb), smoke-testing, statistics processing
- monitoring
- ops maintenance server
- uat
- all prod stuff on one server
- staging
- mirror of the prod servers, where deployment is tested and verified
- production
-
(all redundant) - load balancers, servers with interface for
registered users, servers for anonymous clients, batch
processing servers, database servers, storage servers
3. Related Perl modules
Why Sys::Path?
$Config::Config{'prefix'}
eq
'/usr'
Sys::Path
supply autoconf style installation directories
say Sys::Path->sysconfdir;
# /etc
say Sys::Path->datadir;
# /usr/share
say Sys::Path->logdir;
# /var/log
say Sys::Path->sysconfdir;
# /home/daxim/local/etc
say Sys::Path->datadir;
# /home/daxim/local/share
say Sys::Path->logdir;
# /home/daxim/local/log
Why Module::Build::SysPath?
Module::Build::SysPath
Build.PL
my $builder = Module::Build::SysPath->new(
module_name => 'Debian::Apt::PM',
configure_requires => {
'Module::Build::SysPath' => 0.13,
},
Module::Build::SysPath
YourApp::SPc
sub prefix { ↲
use Sys::Path; ↲
Sys::Path->find_distribution_root(__PACKAGE__); ↲
};
sub localstatedir ↲
{ __PACKAGE__->prefix }; ↲
sub sysconfdir ↲
{ File::Spec->catdir(__PACKAGE__->prefix, 'etc') };
sub datadir ↲
{ File::Spec->catdir(__PACKAGE__->prefix, 'share') };
sub cachedir ↲
{ File::Spec->catdir(__PACKAGE__->localstatedir, 'cache') };
sub logdir ↲
{ File::Spec->catdir(__PACKAGE__->localstatedir, 'log') };
...
Module::Build::SysPath
YourApp::SPc after install
sub prefix { ↲
use Sys::Path; ↲
Sys::Path->find_distribution_root(__PACKAGE__); ↲
};
sub sysconfdir {'/etc'}
sub datadir {'/usr/share'};
sub cachedir {'/var/cache'};
sub sharedstatedir {'/var/lib'};
Sys::Path Module::Build::SysPath bonus => /etc
CPAN::Mini::Inject
- CPAN::Mini
- +custom tarballs
-
mcpani --authorid LOCAL --add \
--discover-packages --file dark-01.tar.gz
CPAN::Mini::Webserver
Pod::POM::Web + trick
CPAN::Patches
- cd Some-Distribution
- cpan-patches list
- cpan-patches patch
- || cpan-patches update-debian
CPAN::Patches::Plugin::Debian
Sum up
- go and break the OS in a bad way
- then rebuild the process step-by-step
- go for multi-server design
What's the vision?
- past - present - future
- have the similar tool for DPAN as CPAN has
- add distribution layer over C|DPAN
- time snapshoting distribution repositories
- make machine ♲ part of a deployment processes
- A/B testing old/new testing killing the wrong attempt
Thanks!
Any answers?