lftp mirror of a directory

open http://mirror.vcu.edu

cd /pub/gnu+linux/centos/6.0/isos

/i386# mirror the 64 bit centos 6
mirror \
-c -e \
–verbose \
.  /opt/iso/centos/6_32bit \
–exclude .torrent \
–exclude .iso \
-i CentOS-6.0-i386-netinstall.iso \
-i CentOS-6.0-i386-minimal.iso \
-i CentOS-6.0-i386-LiveCD.isocd /pub/gnu+linux/centos/6.0/isos/x86_64mirror \
-c -e \
–verbose \
. /opt/iso/centos/6_64bit \
–exclude .torrent \
–exclude .iso \
-i CentOS-6.0-x86_64-netinstall.iso \
-i CentOS-6.0-x86_64-minimal.iso \
-i CentOS-6.0-x86_64-LiveCD.iso#CentOS-6.0-i386-LiveDVD.iso
#CentOS-6.0-i386-bin-DVD.iso
#CentOS-6.0-i386-minimal.iso
#CentOS-6.0-i386-netinstall.iso

#get -c -e \
#       /pub/gnu+linux/centos/6.0/isos/x86_64/CentOS-6.0-x86_64-minimal.iso \
#       -o /opt/iso/centos/6_64bit
#       /pub/gnu+linux/centos/6.0/isos/x86_64/CentOS-6.0-x86_64-netinstall.iso \
#       -o /opt/iso/centos/6_64bit
#       /pub/gnu+linux/centos/6.0/isos/x86_64/CentOS-6.0-x86_64-LiveCD.iso \
#       -o /opt/iso/centos/6_64bit

exit

  • Examples:#lftp -e “set net:timeout 10; cd IN; mput FILENAME ; ls -l;bye” -u “USERNAME”,”PASSWORD” HOSTNAME
    # verify file size:
    #lftp -e “set net:timeout 10; cd IN; ls -l;bye” -u “USERNAME”,”PASSWORD” HOSTNAME
    # remove file
    #lftp -e “set net:timeout 10; cd IN; rm FILENAME ;bye”-u “USERNAME”,”PASSWORD” HOSTNAME

Resources:

  • http://www.cyberciti.biz/faq/lftp-mirror-example/
  • russbrooks.com/2010/11/19/lftp-cheetsheet
  • http://www.kenhess.com/4436.html

Winbind Ties Linux and Windows Sign-Ons Together

Cached from: http://www.enterprisenetworkingplanet.com/netos/article.php/3499006/Winbind-Ties-Linux-and-Windows-SignOns-Together.htm

You keep hearing “Linux is like way cool! Use Linux!” Linux is cool, and even useful, but migrating from other platforms or integrating Linux hosts into an existing network takes a bit more work and knowledge than giving a careless wave of your hand, which is a minor detail that seems to escape the attention of enthusiastic Linux evangelists. Most sysadmins do not have the luxury of starting from scratch, and must make do with existing setups of varying (in)sanity and (il)logic.

The premier program for integrating Windows and Linux is Samba, which ace admins already know and love. Samba can be a cross-platform print and file server, a primary domain controller for a Windows LAN, and even a full member of an NT or Active Directory domain. The difficulty with running mixed Linux and Windows networks is managing user and group accounts and logins. The two platforms manage them in very different ways, which makes it difficult to integrate the two. A common method is to maintain two duplicate sets of users, groups, and passwords, which of course is less than ideal. (The word “sucks” can be confidently applied to this scenario.)

Fortunately the brainiacs behind Samba invented winbind to provide a unified logon, thus saving overworked admins from silliness like doing everything twice, and users from the horrors of trying to track what they are logging into, and which login to use. Winbind lets a Linux box become a full member of a Samba, Windows NT4 or Active Directory domain, and view Windows users and groups as Linux users and groups. All user and group queries from a Linux box are resolved by the domain controller.

Winbind is ideal for admins who wish to add Linux workstations or servers to an existing Windows domain. This allows a graceful introduction with a minimum of hassle. Servers and workstations slide right in without troubling users or bothering pointy-haired bosses.

You should also use winbind when you have hosts that are not members of the domain accessing a Samba or Windows domain. This is an important step to prevent unauthorized access from same-named foreign user accounts. For example, without winbind user Carla who is not a member of the domain will be able to access the files of user Carla who is a member of the domain. This, of course, is bad. Winbind does not allow this to happen; the foreign Carla will be given a different SID (security identifier) and so will not be able to get into the wrong files.

If you’re using a simple peer network without user authentication, don’t bother with winbind. Just for you bullet-point aficionados, here is a summary of what winbind does:

  • Authenticates users
  • Manages passwords
  • Allows Linux users to use Windows domain resources as though they were native Linux resources
  • User and group ID allocation

Underlying Protocols

Winbind makes use of the both the Windows NT RPC (remote procedure calls) and the native protocols of Active Directory. The Samba team received no assistance from Microsoft in decoding these calls, but somehow through persistence and tireless effort captured and decoded the signals over the wire. Moral: if interoperability and customer service are your desire, stick with the free/open source world.

For a NT4/Samba domain, you need Samba 3, winbind, NSS (name service switch), and PAM (pluggable authentication modules). For Active Directory you need Samba 3, winbind, LDAP, and Kerberos.

Using PAM allows authentication and password management to take place on the domain controller. PAM also lets the stern, controlling sysadmin set different authentication policies for different situations, such as for applications, or for users from different locations.

NSS provides a mechanism for hostnames, mail aliases, and user data to be resolved from various sources. Active Directory uses LDAP and Kerberos natively, which Linux can also do.

Windows uses RIDs (relative identifiers) for users and groups. Winbind converts these to Linux user and group numbers in a persistent database file called winbind_idmap.tdb. This is not used when LDAP is used. Additionally, to speed up performance, winbind uses caching. The cache responds to requests, rather than hitting the domain controller for every request.

Requirements

Samba and Winbind also work for just about any Unix variant. We’ll use Linux in this series.

Collecting the necessary bits will vary depending on which Linux distribution you are using. You definitely want the latest stable version of Samba. Debian’s Samba package includes everything you need, including support for PAM already compiled in. Most RPMs do the same. If you want to build from sources, see Resources.

If you are modifying an existing Samba server, back up everything! Back up smb.conf, /etc/pam.d, and /etc/nsswitch.conf. Messing up your PAM configuration means you may not be able to log in at all, so you must also have a rescue disk, such as Knoppix, at hand. Come back next week to learn how to configure all these things to make winbind do the heavy lifting for you.

Resources

Integrate Linux with Active Directory using Samba, Winbind, and Kerberos

Cached from: http://community.spiceworks.com/how_to/show/445

This is the summary of my experience setting up a Linux machine to become a member of an existing Active Directory domain.

Last year I was new to an organization that has an unhealthy affinity for Dell. Anyway, we needed some more storage space, so my solution was to build a server from parts and use Centos 5.1 and Samba to share files with the Windows domain. I wound up with a 3U file server with a 12 TB Raid 6 array with a hot spare (redundancy is serious business) for just shy of $4,000. Given the cost of a similar solution from Dell, major brownie points for me. But, I digress. Here’s the meat and potatoes of getting a linux box to play well with an AD environment.

I hope this helps someone!

1.

Get your linux box configured, with the relevant packages installed.

So, you’ve got your server/workstation up with your favorite flavor of linux installed, and it’s time to join the Windows domain. For this, we’ll be needing samba and kerberos. Most distros come with samba installed, but it’s best to go ahead and grab the newest version either from your distro’s repositories or the samba website itself. Also, make sure you have the krb5 packages installed.
2.

Time synchronization…

AD is very picky about the time matching during authentication, so you’ll need to point the ntpd process to a server on your network. A domain controller is a good choice.

On redhat flavored linux (CentOS, RHEL, and maybe SuSE, I’m not sure on that one) you can configure NTP without editing a .conf file like so:

ntpdate HOSTNAME

For debian flavored linux, edit /etc/ntp.conf with your favorite text editor. Real men use vi. You’ll see a servers section; just replace what’s there with one or more NTP servers on your domain, like so:

server HOSTNAME iburst dynamic

Now, restart the NTP service like so:

service ntp restart

or

/etc/init.d/ntp restart

or

/etc/rc.d/init.d/ntp restart

depending on your particular brand of *nix.

Make sure it’s working with the following command:

ntpq -p

You’ll see some output that should include the NTP server you pointed it to, and some stats.
3.

Edit /etc/hosts

Add this line to /etc/hosts for each domain controller:

xxx.xxx.xxx.xxx adserver.yourdomain adserver
4.

Edit /etc/krb5.conf

Edit /etc/krb5.conf to look something like this:

[libdefaults]
ticket_lifetime = 600
default_realm = YOURDOMAIN
default_tkt_enctypes = des3-hmac-sha1 des-cbc-crc
default_tgs_enctypes = des3-hmac-sha1 des-cbc-crc
[realms]
YOURDOMAIN = {
kdc = ip of you ads server
default_domain = YOURDOMAIN
}
[domain_realm]
.yourdomain = YOURDOMAIN
yourdomain = YOURDOMAIN
[kdc]
profile = /etc/krb5kdc/kdc.conf
[logging]
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmin.log
default = FILE:/var/log/krb5lib.logog
5.

Test kerberos authentication

Enter the following at the shell to test kerberos authentication:

kinit username@DOMAIN

It will prompt for a password, and if all is well, return you to the prompt.

Use the command klist to verify you received a ticket. If you have a ticket, then you’re doing great. If not, double check your /etc/krb5.conf file.
6.

Configure Samba and Winbind to be a domain member.

Almost done. Now we need to edit the /etc/samba/smb.conf file. I’ll include the important parameters. Your smb.conf file should look something like this:

[global]
workgroup = domainname
password server = hostname of domain controller
wins server = IP of wins server
realm = DOMAIN
security = ads
idmap uid = 16777216-33554431
idmap gid = 16777216-33554431
template shell = /bin/bash
winbind use default domain = false
winbind offline logon = false
winbind separator = + <<very important, as the default \ character does strange things in unix/linux.
allow trusted domains = Yes <<if you have them

Those are the important bits, but you’ll find that there are hundreds of valid parameters for the samba config file. Explore them; it’s a very powerful program.
7.

Tell linux to allow winbind to handle authentication.

Edit your /etc/nsswitch.conf to look something like this:

passwd: compat winbind
shadow: compat
group: compat winbind
8.

Moment of truth: Join the domain.

Once the /etc/samba/smb.conf file is properly edited, enter the following at the shell:

testparm

It gives you the rundown of your samba config file, and will let you know if something is wrong. If all is well, it’s time to start the smb and winbind services, like so: (depending on *nix flavor)

service smb restart
service winbind restart

or

/etc/init.d/smb restart
/etc/init.d/winbind restart

or

/etc/rc.d/init.d/smb restart
/etc/rc.d/init.d/winbind restart

If they both come back up fine, lets move to joining the domain, like so:

net ads join -U DOMAIN+username%password

Then test the join using:

net ads testjoin

If it reports “Join is OK”, the test winbind:

wbinfo -u <lists all of your AD users>
wbinfo -g <lists all of your AD groups>

If it works, your linux box is now integrated into the AD domain.
9.

Lastly, configure the smb and winbind services to start automatically

Every distro has a different way of doing this, so I won’t delve into too much detail. Just have a google on it; theres a wealth of information out there.

Conclusion

That should do it. If you run into errors, I may be able to help you. I had a bumpy road getting this up and running, but in the end this is what worked.

Enjoy, and again, I hope this helps someone.

Counting items with mysql and regex in the group by statement

Page Cached from:
http://www.cruzinthegalaxie.com/counting-items-with-mysql-and-regex-in-the-group-by-statement/

Yesterday I needed to do a little report for the boss but the data that I needed to group together wasn’t exactly the same. I hit up the “Goog” for ways to use regular expressions in a count query, but that is kinda tricky to find so this is what I was able to come up with….

So let’s say you had a table where you were attaching the landing url a site visiting is entering your website on, and you wanted to track that landing url the whole way to the end of some web based procedure. Of course certain urls will be different but some could be common for the purpose of counting them together on a report. In a practical example, lets say you are doing pay-per-click traffic with Google and Yahoo. Both of these providers use unique ids in the url to help distinguish between multiple clicks from the same person (so you end up with many urls that are different but mean the same thing). And of course you were a good little developer when you decided to send Google traffic to /google/ and Yahoo to /yahoo/!

With that premise and the following fields in a database table called “signup”, you can group the unique Google and Yahoo landing pages together with a simple regular experssion.

signup table

+----------------------+
| Field
+----------------------+
| id
| signup_date
| landing_page
| etc....
+----------------------+

SELECT
DATE_FORMAT(signup_date,’%m-%d-%Y’) as signup_date,
count(landing_page) as signup_count,
IF(landing_page REGEXP ‘^/(google|yahoo).*’,SUBSTRING(landing_page,1,7),landing_page) as lp
FROM signup
WHERE
1=1
AND signup_date >= ‘” . $istartdate . “‘
AND signup_date <= ‘” . $ienddate . “‘
GROUP BY DATE_FORMAT(signup_date,’%m-%d-%Y’), IF(landing_page REGEXP ‘^/(google|yahoo).*’,SUBSTRING(landing_page,1,7),landing_page)

If that makes no sense whatsoever…. awesome!

When I make a report based on grouping data by day, I select the date for each row, then the count of the column I am looking at, and in this example the field that I am grouping to look at. (This query is sitting behind a web form so the php variables for start and end date are passed in.) In the GROUP BY clause, if you specify the first grouping as the formatted date, you can base the format on the way you are outputting the information on the report. In this example I am doing the report by day. If you wanted to grab the values by month, you would just change the date format pattern from this: ‘%m-%d-%Y’ to ‘%m-%Y’. The tricky part in this query was I need to have one grouping condition that would pull together all the like landing page urls as each value. I chose to use an if statement IF(Condition,true,false) where the condition was a regular expression pattern. Using the condition = landing_page REGEXP ‘^/(google|yahoo).* ‘ tells MySQL that IF the landing_page starts with ‘/google’ or ‘/yahoo’ return the true value, otherwise return the false value. I was not able to return the entire value, so using SUBSTRING will chop off the characters that you want and ignore the rest. If the pattern fails to match it will return the normal value for landing_page. And that is all there is to it.

Let me know if that was helpful to anyone!