Wednesday, November 30, 2011

Installing Redis Database as Windows Service (Redis.io DB) : Issues and Workarounds

Run Redis.io DB as Windows Service

I recently started experimenting with Redis database (a NoSQL DB) as an alternative to SQL-Server for certain development requirements.
"Redis is an open source, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets."
Since I do most of my development under Microsoft Windows, I was hoping to run my Redis instance on my Windows7 x64 Pro development desktop; more specifically, I wanted to run Redis as a Windows service.

Since Redis natively targets Linux/Unix environments, I went searching to see if there was a Windows port of the Redis database project that included the ability to run it as a service.  I found two open-source projects that, when combined, allow me to run Redis on Windows as a Service.


Running Redis DB on Win-7 x64
Installation Notes


The first thing I did was acquire a Windows port of the Redis server. I ended up using this project's compiled-version: https://github.com/kcherenkov/redis-windows-service. The project is described as:
"Windows 32 and x64 port of Redis server, client and utils. 
It is made to be as close as possible to original unix version."
That "close as possible" statement mainly refers to how Redis commands that would (under Unix) rely on fork() to perform background operations are implemented as foreground operations (in Windows port).  But, for purposes of my software development and testing, this would suffice.  I can run my "production" instance of Redis on one of my Linux virtual machine instances (in particular, I have it running on OpenSuse 12.1 x64).

Get the Redis.io for Windows Build
To begin with, download the actual Redis.io (for windows) builds from here: https://github.com/dmajkic/redis/downloads (In my case, I selected the latest x64.zip build, which was redis-2.4.2-win32-win64-fix.zip).

Within that zip-archive, you will see two sub-directories: one is "32bit" and the other is "64bit".  Those zip file directories include the redis-server.exe and redis-cli.exe files (and redis.conf configuration file, etc).  Simply copy the contents of the archive's "64bit" (or 32bit) directory into the chosen directory where you will run Redis from. For example, I placed the x64 files into c:\Redis\

Theoretically, I simply needed to get the redis-server.exe running as a service now...


How to Install Redis DB as a Windows Service:
Win-7 x64 Installation Notes

OK, I have the Redis for Windows executables in my c:\Redis\ directory. Now it is time to get this database running as a Windows Service.  I found one such project that appeared active enough to merit consideration: (link) Run Redis as Service on Windows project on GitHub.

You need to download the compiled executable (RedisService.exe), which is available as a rather small (7 or 8KB) file on the "downloads" page for the project, and place it in your Redis directory.

Note: you may wish to reference this Microsoft site: using SC to create a service if you wish to understand in more detail what the upcoming commands I discuss are doing.

Although you may experience issues (as I will discuss next), you are now supposedly ready to install and start the RedisService.exe (from the command-line in a Windows console window) with the following command (note: alter the "Redis242" service-name to whatever makes sense for you as a process-label; also, change c:\redis portions to whatever directory location you chose):


sc create Redis242 start= auto DisplayName= Redis242 binpath= "\"C:\Redis\RedisService.exe\" C:\Redis\redis.conf"

IF the above statement *appears* to work, the service may or may not start when you execute the following: 

sc start Redis242


But, if you experience some of what I did, the service may be failing for what I will call "hidden" reasons...

Fix Redis Windows Service Problems
and Potential Issues to Workaround

What I discovered with this RedisService.exe windows-service for Redis is that it is quite typical for open-source code: it makes a lot of assumptions and does little to provide proper dependency-testing and meaningful error-condition notification.

When you create the service (per above code: sc create ...) and/or try to start the service (using sc start) it may appear to just "hang" or otherwise take a very long time to attempt to start prior to failing with timeout errors.

The reason for redis-windows-service failing to start properly will be obfuscated, and here are some reasons why:
  • Starting the Service will throw 1053 (timeout) errors without indicating why, but one possible failure reason is that you must have the .NET Framework 4.0.30319 installed for this service to work.
  • Next, depending on your security setup (like my Windows-7 Pro security settings), you may need to tell Windows Firewall that it is OK for this process to act on your local network.  The easiest way to do this is run the redis-server.exe from the command-prompt and allow it access (to local network, through Firewall) when prompted.
  • Next, if you attempt run the redis-server again, you may see another (otherwise hidden) issue in that the executable is not from a "trusted source" or such: again, this issue can be resolved by choosing to allow this un-trusted process to run when provided the option.

After resolving this list of potential issues, you should be able to execute the sc create command and then perform an sc start redis242 (or whatever name you gave the service), to start the Redis Windows service and no longer experience a 1053 error due to timeouts caused by hidden reasons.

Redis Windows Service-Shutdown Problems

Note: there are problems with shutting down this service!  So far, the only way I have found to truly stop it is to reboot my system.  Also, when attempting to delete the service (sc delete redis242 or such), you will not be able to truly delete it as long as any Windows Service-Manager windows are open.

Once the RedisService.exe is installed and actually working, even "sc delete" requires a system reboot to take effect, since you can not otherwise truly stop the service.

The good news...
Although this service is problematic (as of when I wrote this tech blog entry), the program will run as a service and the client (redis-cli) can now be executed against the service-induced redis-server to test SET/GET of keys, etc.

If you are interested in accessing Redis.io from JavaScript, you may want to read my blog about NPM (Node Package Manager) where my example for installing a Nodejs module used a node-redis module.  I am able to access my Redis (NoSQL) DB from both Linux and Windows versions of Nodejs via the node-redis module's functionality from within Javascript.

Tuesday, November 29, 2011

NPM (Node-Package-Manager) Package Install Error

NPM (Node-Package-Manager) Package Install Errors:
Misleading Error Messages

I recently found myself encountering some nearly useless NPM errors when I was trying to install the redis-node package (i.e., Redis.io database connectivity layer for Nodejs), and it did not instantly occur to me why I was seeing error messages during NPM's archive-unpacking operation, especially as the Node-Package-Manager was dumping out a list of errors that made little sense.

Under Linux, NPM install was producing errors that implied I did not have permission to the tmp / temporary directory or directories the package was being unzipped/unpacked into for installation from (i.e., the temporary location the tarball / tar.gz file would be unpacked in).  Under Windows, the same NPM install (under git-bash window) produced different errors that made it appear the downloaded .git package could not be unpacked for god knows what reason, all hidden in a massive error dump that had nothing to do with the REAL reason for the error.

Well, I figured out why the errors occurred (as detailed in this blog entry), and I also learned that this NPM software is a perfect example of what I consider a non-user-friendly interface in that it presents the user with all sorts of completely meaningless error messages in the event of very simple-to-detect issues.  Bottom line: the node package manager is written by geeks, for geeks.  This is not "enterprise grade" software by any means, as it does not have robust error-detection/traps/messages.  In fact, the error-traps it does employ seem to mislead more than assist: this is simply poor design.  (note: I give credit to ANY open-source effort like this though, and I understand why people focus on other functionality vs. "usability").

Node Package Manager Windows Behavior and NPM Error Messages due to using wrong .git file URL

I have been using Node (Nodejs) for "server-side javascript" development on a Windows 7 Pro x64 machine, with Node "installed" as simply the Node.exe download placed in a local directory of c:\node

From within that directory (using the git-bash terminal window on Windows that was installed with the Git SCM tool version 1.7.7.x Windows .exe installer), I executed the following command:

c:\node>node ./npm install -g https://github.com/mranney/node_redis.git

...but, that produces the following error dump. Can you see why? It was all too clear to me a bit later...
c:\node>node ./npm install -g https://github.com/mranney/node_redis.git


npm ERR! couldn't unpack C:\TEMP\npm-1322598499268\1322598499268-0.7676450146827847\tmp.tgz to C:\TEMP\npm-1322598499268\1322598499268-0.7676450146827847\contents
npm ERR! Error: ENOENT, no such file or directory 'C:\TEMP\npm-1322598499268\1322598499268-0.7676450146827847\contents\package\package.json'
npm ERR! Report this *entire* log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <npm-@googlegroups.com>
npm ERR!
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "node" "c:\\node\\npm" "install" "-g" "https://github.com/mranney/node_redis.git"
npm ERR! cwd c:\node
npm ERR! node -v v0.6.3
npm ERR! npm -v 1.0.105
npm ERR! path C:\TEMP\npm-1322598499268\1322598499268-0.7676450146827847\contents\package\package.json
npm ERR! code ENOENT
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     c:\node\npm-debug.log
npm not ok

Notice the URL I mistakenly specified for the install package... THAT is what causes the error.  Yes, something as simple as copying and pasting the incorrect URL from the project's github page will lead to this tragic error-dump.

This is where I went wrong:

...the default URL shown in that textbox on Github (image above) is for the HTTP version of the URL.  I had accidentally copied that and pasted to my command prompt for my "npm install" command, where instead I needed to do this:


Now I could copy and paste the proper git:// prefixed URL address of the node_redis.git package that I wanted to install with NPM.

c:\node>node ./npm install -g git://github.com/mranney/node_redis.git

...which produces the simple one-line output as a result of a "successful" package install:

redis@0.7.1 c:\node\node_modules\redis

Ah, that is better!

Now, on to what this same issue presents like under Linux...

Node Package Manager Linux Behavior and NPM Error Messages due to using wrong .git file URL

I have been using Node (Nodejs) under OpenSuse 12.1 x64 KDE.  When I first encountered this issue under Windows, I quickly jumped over to my Linux VMware virtual-machine to see if for some reason it was a Windows-implementation-only issue (since, Node and NPM have been more mainstream on Linux, and the NPM under Windows is considered "experimental" yet).

This quick test under Linux helped me see the error in my ways quickly, since I encountered a similar mess of misleading error-messages spewing forth from the node-package-manager when I copied and pasted my npm install command to Linux and executed it:

~/node> node ./npm install -g https://github.com/mranney/node_redis.git

Yes, it failed similarly to the Windows NPM version, but with an error message (pasted here) that sure made it appear like the the "tar" (unpack) command failed for some inability to work with the temp .tgz file created as part of the install process...


npm ERR! Failed unpacking /tmp/npm-1322603639405/1322603639405-0.840085425414145/tmp.tgz
npm ERR! couldn't unpack /tmp/npm-1322603639405/1322603639405-0.840085425414145/tmp.tgz to /tmp/npm-1322603639405/1322603639405-0.840085425414145/contents
npm ERR! Error: `tar "-zmvxpf" "/tmp/npm-1322603639405/1322603639405-0.840085425414145/tmp.tgz" "-o"`
npm ERR! failed with 2
npm ERR!     at ChildProcess.<anonymous> (/home/mike/node/npm/lib/utils/tar.js:217:20)
npm ERR!     at ChildProcess.emit (events.js:70:17)
npm ERR!     at maybeExit (child_process.js:359:16)
npm ERR!     at Process.onexit (child_process.js:395:5)
npm ERR! Report this *entire* log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <npm-@googlegroups.com>
npm ERR! 
npm ERR! System Linux 3.1.0-1.2-desktop
npm ERR! command "node" "/home/mike/node/npm" "install" "-g" "https://github.com/mranney/node_redis.git"
npm ERR! cwd /home/mike/node
npm ERR! node -v v0.6.2
npm ERR! npm -v 1.0.106
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /home/mike/node/npm-debug.log
npm not ok


Having performed successful node module installs with NPM under my Linux host already, I knew that I must be missing something obvious.  And, without any help at all from those meaningless error messages, I finally saw that I pasted the "https://" address of the github node module address instead of the git:// version.  Fixing this mistake, the module installed just fine under both Linux and Windows.

This revelation (about my mistaken "https" vs. "git" prefix), coupled with my frustration at misleading NPM error messages, led to the writing of this blog in case anyone else runs into this issue and gets misled by utterly meaningless error messages.

I hope someone working on the NPM (package manager for node) will eventually make time to implement proper condition-testing (and meaningful error-reporting) to test the format of command-line parameters as important as the actual package/module URL if the format makes such a tremendous difference in results.  Had this software simply tested the URL for a valid format, and told me "sorry, you must use git:// prefix" or suggested another equally effective alternative, that would be ideal and would save myself, and surely others, from wasting time with misleading error messages.


Redis (Redis.io) Database via Node

As an aside, I can report that I am able to access my Redis (NOSQL) DB from both Linux and Windows versions of Node via the node_redis module's functionality from within Javascript.  I hope to add more blog material about my experiences with installing and using Redis with Node at a later date.