kengz

npm vs RubyGem vs PyPI

If you’re a Node developer, then you know how spoilt we are (all hail the almightly babysitter npm).

Let’s not talk about languages, because I firmly believe that languages are designed to suit certain needs, so I don’t take the “which language is better” argument.

Instead, I’d love to talk about packaging - because it’s so central to modern development. A good packaging manager is crucial to foster a good open source standard in a community.

This week I try to find out if we can do stuff the npm-way with the package managers of Python and Ruby, i.e. how do you publish a proper package there.

npm

Here’s a sample npm file structure that I wrote. Of course it’s more well polished than the two below cuz I’m mainly a Node dev.

If you head over to the site, it’s pretty damn active; in fact it’s the world’s largest package manager. People npm install a lot.

It has:

• a nice search interface
• proper package page sourced from its Github’s README.md, with details on installation, usage, etc.
• a side bar with relevant stats

Let’s quickly go over how we work with npm.

1. Create a Github repo with README.md, LICENSE, .gitignore, and clone it.
2. Write your module under lib/, and import the needed methods to one point of entry index.js.
3. At root, run npm init to setup the package.json, which will take care of everything nicely using simple JSON format.
4. Run npm publish and voila. Your new package is up and available immediately. Just do npm i <package> to install.

At the time of writing, npm has about 800Mil downloads per week. That’s more than all the RubyGem downloads (700Mil) ever. This is the magic when doing a thing is so effortless (in npm), everyone does it without a second thought. I remember when I started as a newbie with npm I went through everything smoothly, thanks to the great README.md most packages have.

RubyGem

Here’s a sample RubyGem file structure that I wrote. For guide, look at DigitalOcean’s.

RubyGem is the npm of Ruby, and the packages are called Gems. It the grand-daddy of modern dev, but there’s a few things that made me cringe. Grand-daddy is old now.

It has:

• a bad search interface: unintelligent, non-suggestive, and worst of all you have to go back to the homepage to search again. What year is this?
• the gem pages are mostly poorly written because there isn’t a robust standard for it in the community
• a side bar with some stats, Okay.

Let’s quickly go over how we work with RubyGem.

1. Create a Github repo with README.md, LICENSE, .gitignore, and clone it.
2. Ahhh do gem install bundler first. At root, run npm init to setup the package.json, which will take care of everything nicely using simple JSON format. run bundle gem <gem> to generate the necessary files. That’s pretty nice. But there’s a nightmarish part: the Gem specs are split across different files: write it in functional Ruby <gem>.gemspec without a single standardized format, source the gem dependencies from Gemfile, and add a Rakefile for publishing your gem. No, there is no standardized package.json to handle all these things for you under a single JSON file. Welcome to the brutal world you spoilt brat.
3. Write your module under lib/<gem>, and import the needed methods to one point of entry index.js. add a version.rb under it, then import the needed methods to one point of entry lib/<gem>.rb.
4. Run ~npm publish and voila~ gem push <gem>-<version>.gem. Oh if you think that’s 3 arguments too long, wait till you see how to use Rake. Whyyy gosh whyyy? But you can do gem install <gem> simply, thank god.

PyPI

Here’s a sample PyPI file structure that I wrote, along with a simple guide.

I love Python, but PyPI gives me cancer. Okay I get it, the user base is different from the sassy Node devs, and who gives a shit about user interface? Ain’t nobody got time for that (except for web developers).

It has:

• the most useless search interface using only simple regex. It floods the results with multiple versions of the same package, so good luck browsing.
• ~proper package page sourced from its Github’s README.md,~ PyPI can’t handle Markdown apparently, so enjoy your plain text formatting. Some important packages actually have details on installation, usage, etc. cuz otherwise you’d be fucking lost - goodluck finding the Github page of the package, even if it exists in the first place!
• a side bar with relevant statsMost serious users are scientific coders and they know exactly what package they’re looking for, so who cares bout stats in that case.

Let’s quickly go over how we work with PyPI. But first, good luck with the fragmentation - Python2 v.s. Python3, virtualenv v.s. pyvenv…

Now that you’re done choosing,

1. Create a Github repo with README.md, LICENSE, .gitignore, and clone it.
2. Write your module under <package>/. Add <package>/__init__.py, and import the needed methods within it.
3. At root, run npm init to setup the package.json, which will take care of everything nicely using simple JSON format.Copy the setup.py from someone else who has written it, and there’s no standardization. Ohh wanna list your dependencies? Do pip install <deps> as you dev, then at the end only specify these dependencies using pip3 freeze > requirements.txt
4. Run npm publish and voila. Nightmare isn’t over. First, register your package name python setup.py register. If successful, then upload your smelly tar balls to PyPI python setup.py sdist upload (who the hell devs with tar files and manual installation today?). Your new package is up and available immediately a century later before you can do pip install <package> to install the just-published version.
5. Btw all the cray involving pip and packages are best done with virtualenv or whatever you choose.

Conclusion

I love all the three languages - Node, Ruby, Python. However, modern dev is so much about open source and external dependencies - you can’t write powerful code efficiently without relying on open source modules. No dev is an island now.

That being said, I love npm, but RubyGem and PyPI kinda made me went “ehmmmm, what year is this?” I’ll still have to use both of them (and cringe every time I do), but I just have to suck it up and stop being a spoilt brat. The world isn’t all nice, you know.

Modern dev practices

Modern development isn’t just about writing code - it’s also about the practice as a whole.

Traditionally, we write those complicated, yucky, half-hearted unit tests, run them once and call it a day. If you’ve done Java and JUnit, you know no one gets excited for that. Worse, this practice simply is decrepit.

Thank goodness, today there are nice tools to help us do things the nice way. In modern development, there are some code qc practices that goes with it! Roughly, we have the list below, with the popular choices. The examples will relate to nodejs because that’s my main dev language, although the same practices are language-agnostic.

• version control: Github
• documentation: JSDoc, dokkerjs
• unit testing: Mocha + Chai
• code coverage: Istanbul, Coveralls.io
• dependency check: Gemnasium
• continuous integration: Travis CI

Oh, and these tools are all free! What an amazing time to be a developer, isn’t it?

Version control

Github is undoubtedly everyone’s favorite for VC because it’s just awesome. I sound bias, but to be fair I’ve used SVN, and still use Bitbucket, and I know how much they suck. Seriously. Github is a great tool, with top notch user interface, robust API for developers, and a huge community. Seriously everyone’s on Github.

Why do I still use Bitbucket, you might wonder? It’s only because they offer unlimited private repos for free, whereas Github only gives you 5. I’m a cheapskate, I know :)

Documentation

I’ve been searching for a great documentation generator that is automated - fully automated. Ideally, you should just have to annotate your source code with the docs.

Then, when you’re done, the generator should compile all the docs into a beautifully styled web page, and pushes it automatically to a hosted site - ideally the Github Pages of the project very own repo.

Dokkerjs does exactly that. I found a guy writing it just when I was about to give up searching and write my own. We ended up finishing it together. An example project is lomath.

Occasionally you want something really light, and since Markdown is a popular format that’s compatible with Github, I use jsdoc-to-markdown too. An example project is neo4jKB.

Unit testing

I know most people don’t like it - I did. However, once you find a nice framework to do it, it’ll be fun and satisfying, trust me. Plus, quality code must be tested, 100%-tested.

I love the Mocha framework, which supports both TDD and BDD (test- and behavior-driven-development). My assertation library of choice for Mocha is Chai. Chai even has a variant that supports Promised-based tests. To see example tests with proper configs, look at neo4jKB. This includes a common.js to share global variables, an asset.js to share test assets, and mocha.opts to specify the dev styles and test reporters.

Code coverage

Unit testing alone isn’t enough because, when you source code gets long, you get lost. You have to make sure you’ve tested every function, conditional statement and branches, and considered every possibility. This is what we call code coverage. Basically, it crawls your source code and indexes all the functions with their possible break points and conditional branches.

Mocha does that a bit through it’s reporter, but there’s a more powerful option, Istanbul. It allows you to more carefully exclude the parts that should not be tested, so you can get up to 100% test coverage without looking sloppy. I use Istanbul for offline, local coverage reporting.

Another option is Coveralls.io, which I use for online, deployed coverage reporting. This is nice because it automatically fetches the built tests from your CI (next up!), and updates a badge on your Github repo. Here’s a sample code coverage of neo4jKB

Dependency check

Today we build many things with open source dependencies. It’s good to keep them always updated, and a way to monitor that is through Gemnasium, which also automatically updates a badge on your Github repo. Here’s a sample dependency check of neo4jKB

Continuous integration

This is the final step where all the above falls into place. The idea of continuous integration is that you develop your code, write the unit tests, push to your Github, it then builds your project on several deployment environments and runs the tests. When all that is done, it pushes the code coverage to Coveralls, and reports the latest build status on a badge on your Github repo.

My favorite is Travis CI, and I’ve used it in many of my projects. Here’s a sample build status of neo4jKB; you can see the Mocha tests there too.

Putting it all together

So, what’s the workflow of a modern developer? You write your source code and annotate it with the documentation in the comments. You can run Dokkerjs to let it push the new beautiful doc to the Github Pages fir you.

Then, you write the unit tests, and run locally to make sure it works fine on your machine. When you’re done, push your source code to Github. Travis CI automatically builds your project and runs the unit tests on your push; this ensures your project works on other platforms too.

The latest build status, code coverage, and dependency status are updated automatically on your Github pages. Altogether, you have a properly maintained repo, with automatic status update on the health and quality of your code. Users of your projects know they can rely on you when all badges are green, and know to watch out when some badges turn red. Neat, isn’t it?

I know this is a short post for such a long topic; there isn’t space to elaborate fully how do you do each thing. But, there are many great tutorials out there, and I hope I’ve given you some good options to start with. Go check them out!

If you like to see some examples, check out the pretty badges at my recent repos neo4jKB and lomath. Happy devving!

Restoring machine setup after an EMP attack

Do you also have multiple machines, and have a habit of wiping/reformatting your machines? One reason people don’t wipe as frequently is because it’s hard to setup a machine like you’re used to.

That used to be me; I’m very OCD about my programming environment. Few months ago I had to wipe many times, and the manual machine setup was def a pain. So I asked for more pain by writing a bash script to automate it once and for all.

It’s quite easy. You’re basically compiling all the commands you type into the terminal into a script, and run the bulk at one go. With the liberty of scripting, you can actually improve the process even further.

I develop mostly on MacOS, but I have to setup a Unix at times too. Being all OCD and with over 30 wipes and refinements, my bash mac_setup script is now well polished - it even checks for the OS (Mac or Unix) and run the corresponding subscript.

Here is the setup script. It checks for the target’s existence then install or update. Overall, it:

• configures the system preferences and security settings
• writes the new .bash_profile
• installs apps via brew Cask
• installs the dev dependencies
• downloads and restores my Sublime setting woohoo!

I typically just boot into my fresh new OS, download the script from my Github and leave it to run. When I come back after 20 minutes, I’m greeted with a familiar machine.

Feel free to fork or clone my setup script repo. Yes the repo title says

Set up your new mac in case of an EMP attack.

because u can recover within 20 minutes. Neat ayy?

Anyway here’s a preview of the script:

#!/bin/bash

#####################################
# This script sets up your machine
# Permission set to user readable & executable
# by: chmod 775 ./setup
# run by right-clicking it
#####################################


#####################################
# Preparing to run
#####################################
echo "
================================================

Welcome to the Mac Setup
by Wah Loon Keng @ Dec 15 2015
for bug reports, contact [ kengzwl@gmail.com ]

This will take under 30 minutes.

================================================
"
# Ask for the administrator password upfront
[ "$UID" -eq 0 ] || exec sudo bash "$0" "$@"
# sudo -v
# prevent sleep for this script; & runs in background
caffeinate -d &
# Keep-alive: update existing `sudo` time stamp until this script has finished
while true; do sudo -n true; sleep 60; kill -0 "$$" || exit; done 2>/dev/null &




#####################################
# System Preference customization
#####################################
# Systems prefs, location at: ~/Library/Preferences
# to read, e.g.:
# defaults read com.apple.dock
# reset if u fucked up:
# defaults delete com.apple.dock; killall Dock
echo "
================================================

Customizing System Preferences

================================================
"

#####################################
# System-wide UI / Behavior
#####################################
# use dark menu bar and Dock
defaults write NSGlobalDomain AppleInterfaceStyle -string Dark

# highlight color = red
defaults write NSGlobalDomain AppleHighlightColor -string "1.000000 0.733333 0.721569"

# Require password immediately after sleep or screen saver begins
defaults write com.apple.screensaver askForPassword -int 1
defaults write com.apple.screensaver askForPasswordDelay -int 0

# ... more

# Installs homebrew
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"


# upgrade and check
brew update && brew upgrade
brew doctor

# install the prereq
brew install git
brew install wget

# homebrew taps
brew tap caskroom/cask
brew tap caskroom/versions
brew tap caskroom/fonts


#####################################
# Install common apps
#####################################
# Installs cask, really vital
brew install caskroom/cask/brew-cask

# Install the basic apps for work
brew cask install google-chrome
brew cask install 1password
brew cask install tunnelblick
brew cask install sublime-text3
brew cask install font-source-code-pro
brew cask install slack
brew cask install spotify
brew cask install shiftit
brew cask install java

# download the vpn configs for tunnelblick
curl -o ~/Downloads/openvpn.zip https://www.privateinternetaccess.com/openvpn/openvpn.zip
unzip ~/Downloads/openvpn.zip -d ~/Downloads/openvpn/
rm ~/Downloads/openvpn.zip

# ... and more

npm script with arguments

npm is a powerful scripting tool to build your projects. It unifies all commands and let you do npm start, npm stop, npm run <script> etc., where the actual commands executed by them are defined under package.json.

Recently I’ve had the need to set some environment variables while doing npm start. npm commands allows you to pass arguments or set the env vars in several ways, but I wasn’t quite satisfied:

• loading an .env file, or prepending like <key>=<value> npm start isn’t elegant.
• npm config set <key> <value>: this is cumbersome af. Ain’t gonna write that!
• npm start --<key>=<value>: really close to what I want! But it actually sets $npm_config_<key> instead of $key.

My use case was to start my bots of different names, say:

• npm start would deploy the default bot,
• npm start --bot=veronica would deploy the other bot named veronica

I fiddled with npm and bash for hours and got it figured out. Basically we use the commands above, internally these would happen:

• $npm_config_bot would be set only if --bot=veronica is passed.
• bash checks if $npm_config_bot is set; if so it sets the $bot=veronica; else it sets the default $bot=jarvis
• then the deploy script runs with the $bot variable.

Below is my snippet from package.json:

{
"scripts": {
    "start": "npm run deploy",
    "deploy": "if [ $npm_config_bot ]; then bot=$npm_config_bot; echo Bot is SET to: $bot; else bot=peppurr; echo Bot is DEFAULTED to $bot; fi; DEPLOY=.keys-$bot forever start --uid $bot index.js"
  }
}

Beware that if the command is split into separate npm scripts, due to the local scoping you’d actually lose the $bot variable. That’s why deploy is such a long command.

Github flavored Markdown in Tinypress

Another hack for Tinypress - if you notice, it uses a different kind of Markdown than the one we’re used to on Github. This can be fixed by changing the ./_config.yml that Tinypress uses, like mine here, which looks like:

excerpt_separator: ""
pygments: true
markdown: redcarpet
markdown_ext:  markdown,mkdown,mkdn,mkd,md
redcarpet:
  extensions: ["tables", "autolink", "strikethrough", "space_after_headers", "with_toc_data", "fenced_code_blocks"]
url: https://kengz.me
title: kengz
tagline: Kengz.me
description: Mein blog
paginate: 5

Happy blogging!