RabbitMQ REST API

Thanh Phu on October 20, 2017

There’s a little known REST API built into RabbitMQ, accessible via the management plugin. Its methods are as follows:

GET PUT DELETE POST Path Description
X /api/overview Various random bits of information that describe the whole system.
X X /api/cluster-name Name identifying this RabbitMQ cluster.
X /api/nodes A list of nodes in the RabbitMQ cluster.
X /api/nodes/name An individual node in the RabbitMQ cluster. Add “?memory=true” to get memory statistics, and “?binary=true” to get a breakdown of binary memory use (may be expensive if there are many small binaries in the system).
X /api/extensions A list of extensions to the management plugin.
X X /api/definitions
/api/all-configuration (deprecated)		 The server definitions – exchanges, queues, bindings, users, virtual hosts, permissions and parameters. Everything apart from messages. POST to upload an existing set of definitions. Note that:

  • The definitions are merged. Anything already existing on the server but not in the uploaded definitions is untouched.
  • Conflicting definitions on immutable objects (exchanges, queues and bindings) will cause an error.
  • Conflicting definitions on mutable objects will cause the object in the server to be overwritten with the object from the definitions.
  • In the event of an error you will be left with a part-applied set of definitions.

For convenience you may upload a file from a browser to this URI (i.e. you can use multipart/form-data as well as application/json) in which case the definitions should be uploaded as a form field named “file”.
X /api/connections A list of all open connections.
X X /api/connections/name An individual connection. DELETEing it will close the connection. Optionally set the “X-Reason” header when DELETEing to provide a reason.
X /api/connections/name/channels List of all channels for a given connection.
X /api/channels A list of all open channels.
X /api/channels/channel Details about an individual channel.
X /api/consumers A list of all consumers.
X /api/consumers/vhost A list of all consumers in a given virtual host.
X /api/exchanges A list of all exchanges.
X /api/exchanges/vhost A list of all exchanges in a given virtual host.
X X X /api/exchanges/vhost/name An individual exchange. To PUT an exchange, you will need a body looking something like this:

{"type":"direct","auto_delete":false,"durable":true,"internal":false,"arguments":{}}

The type key is mandatory; other keys are optional.When DELETEing an exchange you can add the query string parameter if-unused=true. This prevents the delete from succeeding if the exchange is bound to a queue or as a source to another exchange.
X /api/exchanges/vhost/name/bindings/source A list of all bindings in which a given exchange is the source.
X /api/exchanges/vhost/name/bindings/destination A list of all bindings in which a given exchange is the destination.
X /api/exchanges/vhost/name/publish Publish a message to a given exchange. You will need a body looking something like:

{"properties":{},"routing_key":"my key","payload":"my body","payload_encoding":"string"}

All keys are mandatory. The payload_encoding key should be either “string” (in which case the payload will be taken to be the UTF-8 encoding of the payload field) or “base64” (in which case the payload field is taken to be base64 encoded).
If the message is published successfully, the response will look like:

{"routed": true}

routed will be true if the message was sent to at least one queue.Please note that the HTTP API is not ideal for high performance publishing; the need to create a new TCP connection for each message published can limit message throughput compared to AMQP or other protocols using long-lived connections.
X /api/queues A list of all queues.
X /api/queues/vhost A list of all queues in a given virtual host.
X X X /api/queues/vhost/name An individual queue. To PUT a queue, you will need a body looking something like this:

{"auto_delete":false,"durable":true,"arguments":{},"node":"[email protected]"}

All keys are optional.When DELETEing a queue you can add the query string parameters if-empty=true and / or if-unused=true. These prevent the delete from succeeding if the queue contains messages, or has consumers, respectively.
X /api/queues/vhost/name/bindings A list of all bindings on a given queue.
X /api/queues/vhost/name/contents Contents of a queue. DELETE to purge. Note you can’t GET this.
X /api/queues/vhost/name/actions Actions that can be taken on a queue. POST a body like:

{"action":"sync"}

Currently the actions which are supported are sync and cancel_sync.
X /api/queues/vhost/name/get Get messages from a queue. (This is not an HTTP GET as it will alter the state of the queue.) You should post a body looking like:

{"count":5,"requeue":true,"encoding":"auto","truncate":50000}
  • count controls the maximum number of messages to get. You may get fewer messages than this if the queue cannot immediately provide them.
  • requeue determines whether the messages will be removed from the queue. If requeue is true they will be requeued – but their redelivered flag will be set.
  • encoding must be either “auto” (in which case the payload will be returned as a string if it is valid UTF-8, and base64 encoded otherwise), or “base64” (in which case the payload will always be base64 encoded).
  • If truncate is present it will truncate the message payload if it is larger than the size given (in bytes).

truncate is optional; all other keys are mandatory.

Please note that the get path in the HTTP API is intended for diagnostics etc – it does not implement reliable delivery and so should be treated as a sysadmin’s tool rather than a general API for messaging.
X /api/bindings A list of all bindings.
X /api/bindings/vhost A list of all bindings in a given virtual host.
X X /api/bindings/vhost/e/exchange/q/queue A list of all bindings between an exchange and a queue. Remember, an exchange and a queue can be bound together many times! To create a new binding, POST to this URI. You will need a body looking something like this:

{"routing_key":"my_routing_key","arguments":{}}

All keys are optional. The response will contain a Location header telling you the URI of your new binding.
X X /api/bindings/vhost/e/exchange/q/queue/props An individual binding between an exchange and a queue. The props part of the URI is a “name” for the binding composed of its routing key and a hash of its arguments.
X X /api/bindings/vhost/e/source/e/destination A list of all bindings between two exchanges. Similar to the list of all bindings between an exchange and a queue, above.
X X /api/bindings/vhost/e/source/e/destination/props An individual binding between two exchanges. Similar to the individual binding between an exchange and a queue, above.
X /api/vhosts A list of all vhosts.
X X X /api/vhosts/name An individual virtual host. As a virtual host usually only has a name, you do not need an HTTP body when PUTing one of these. To enable / disable tracing, provide a body looking like:

{"tracing":true}
X /api/vhosts/name/permissions A list of all permissions for a given virtual host.
X /api/users A list of all users.
X X X /api/users/name An individual user. To PUT a user, you will need a body looking something like this:

{"password":"secret","tags":"administrator"}

or:

{"password_hash":"2lmoth8l4H0DViLaK9Fxi6l9ds8=", "tags":"administrator"}

The tags key is mandatory. Either password or password_hash must be set. Setting password_hash to “” will ensure the user cannot use a password to log in. tags is a comma-separated list of tags for the user. Currently recognised tags are “administrator”, “monitoring” and “management”.
X /api/users/user/permissions A list of all permissions for a given user.
X /api/whoami Details of the currently authenticated user.
X /api/permissions A list of all permissions for all users.
X X X /api/permissions/vhost/user An individual permission of a user and virtual host. To PUT a permission, you will need a body looking something like this:

{"configure":".*","write":".*","read":".*"}

All keys are mandatory.
X /api/parameters A list of all parameters.
X /api/parameters/component A list of all parameters for a given component.
X /api/parameters/component/vhost A list of all parameters for a given component and virtual host.
X X X /api/parameters/component/vhost/name An individual parameter. To PUT a parameter, you will need a body looking something like this:

{"vhost": "/","component":"federation","name":"local_username","value":"guest"}
X /api/policies A list of all policies.
X /api/policies/vhost A list of all policies in a given virtual host.
X X X /api/policies/vhost/name An individual policy. To PUT a policy, you will need a body looking something like this:

{"pattern":"^amq.", "definition": {"federation-upstream-set":"all"}, "priority":0, "apply-to": "all"}

pattern and definition are mandatory, priority and apply-to are optional.
X /api/aliveness-test/vhost Declares a test queue, then publishes and consumes a message. Intended for use by monitoring tools. If everything is working correctly, will return HTTP status 200 with body:

{"status":"ok"}

Note: the test queue will not be deleted (to to prevent queue churn if this is repeatedly pinged).

This can probably be used to build a library that monitor RabbitMQ nodes and automatically select the best node to connect to

Management task: Remote shutdown utility

Thanh Phu on October 17, 2011

I run a simple web server to serve as a storage medium for my mobile devices (I don’t like using dropbox or other online storage solutions because they don’t offer much space and they hogs up bandwidth unnecessarily). Before I go to sleep I usually shut down the server to save energy, but in order to do so, I have to have another computer turned on to perform remote access into the server and shut it down via command line. Occasionally, I would turn off my work computer before I remember that I want to shut down the server and I’m too lazy by then to turn on my work computer again.

That’s why I wanted an application that is capable of shutting down Windows on my behalf. The scenario is like this: because I can access FTP service from my phone, I will create a file, namely ‘shutdown’; when the server sees this file, it will delete the file and initiate shutdown. Quite simple.

To shut down, I use Windows management interface (System.Management namespace in C#)

        static void Shutdown()
        {
            ManagementBaseObject mboShutdown = null;
            ManagementClass mcWin32 = new ManagementClass("Win32_OperatingSystem");
            mcWin32.Get();

            // You can't shutdown without security privileges
            mcWin32.Scope.Options.EnablePrivileges = true;
            ManagementBaseObject mboShutdownParams =
                     mcWin32.GetMethodParameters("Win32Shutdown");

            // Flag 1 means we want to shut down the system. Use "2" to reboot.
            mboShutdownParams["Flags"] = "1";
            mboShutdownParams["Reserved"] = "0";
            foreach (ManagementObject manObj in mcWin32.GetInstances())
            {
                mboShutdown = manObj.InvokeMethod("Win32Shutdown",
                                               mboShutdownParams, null);
            }
        }

From here

And the rest is just a loop to check if the file exists

        /// 
        /// The main entry point for the application.
        /// 
        [STAThread]
        static void Main()
        {
            Application.EnableVisualStyles();
            Application.SetCompatibleTextRenderingDefault(false);

            while (true)
            {
                if (!System.IO.File.Exists("D:\Shutdown"))
                {
                    Thread.Sleep(500);
                }
                else
                {
                    System.IO.File.Delete("D:\Shutdown");
                    Shutdown();
                    return;
                }
            }
        }

That’s it, I run the application and it will wait until I create a file named ‘shutdown’ in D:, it will perform shutdown and quit

Here’s the sample code and binary

Notes on installing and configuring Windows Software Update Service under offline model for isolated network

Thanh Phu on October 14, 2009

Executive summary

Due to strict requirements of security for finance businesses, a finance company’s network should be isolated from outside environment. This has the effect of shielding making the entire software infrastructure less prone to software invulnerabilities and thus external attacks. However, this does not eliminate the need for software updates since the network’s operation can still be disrupted by internal sabotage or worm infection from careless users.

Since large financial institutions often have management workflows, updates are subject to prior approval by the management and not all updates maybe approved. The process of maintaining software updates manually, which includes downloading, tracking, approving and installing’s cost could escalate as vulnerabilities are discovered over time and software enhancements are made.

As a major software manufacturer, Microsoft is aware of this issue and have developed Windows Software Update Service (WSUS) as a solution for the update management problem. WSUS allows for approval, mass install, provides a central configuration point and thus reduces the maintenance cost.

This document tried to summarize the knowledge needed for installing and applying WSUS’s update model to an isolated network. This includes:

1. The requirements and how to install WSUS under the isolated model.

2. Configuring WSUS and the approval workflow with WSUS.

3. Synchronize update databases between computers.

4. Configure clients to receive intranet software updates.

Infrastructure requirements

image2

Figure 1 A typical corporate intranet

The picture above describes a typical “closed-circuit” corporate network, which connects computers within a branch and branches together. This network is isolated from outside environment (e.g. the internet or other non-corporate affiliated networks).

Two servers, disconnected model

Because the update server must be connected to the internet somehow (to receive updates from the root update server – Microsoft’s), the safest model for this application is to utilize two server – one to receive updates from outside and one to propagate updates inside the intranet. Data would be manually synchronized between the two machines by various means like tapes, external HDDs or an ad-hoc network. Tape and HDD synchronization theoretically is the most secure option, since no connection is required between two machines and thus, makes data leaks and network penetrations almost impossible (data can only go in and can’t go out). Also, update package’s integrity is always verified with digital signatures from Microsoft so they can’t be compromised.

image3

Figure 2 Two servers model

Package approvals can be made on the internal server. Subsequent package updates from the external server will not affect existing approvals.

Although being the most secure option, this model is costly to deploy:

  • It requires an additional server to receive updates from the internet.
  • It requires additional data containing medium (tapes, HDDs) to synchronize the servers securely.
  • It takes time to synchronize (updates are usually large – to the extent of terabytes and external storage media is usually slow)

This technical note is focused primarily on how to configure and make servers perform this role.

One server, switching connection model

image005

Figure 3 Switching connection model – update phase

image006

Figure 4 Switching connection model – distributing phase

To avoid the cost of an additional server and updating time, this one-server model can be used. The update server could switch between two connections: one to the internet (to download updates) and one to the internal network (to distribute updates). The switch could be made at anytime because WSUS itself could handle partial downloads.

With this model, penetrations and leaks may occur if a malicious party could install a drone in the update server to execute predetermined commands or probe for data from the internal network. Therefore t is advised that the update server be carefully firewalled and allow only the update service to access the network interface.

Also, the update server must be dedicated to the role can can’t perform other server roles since it could be disconnected from the internal network at any time.

One server, always on model

image007

Figure 5 One server, always on model

This model is similar to the one server, switching model except that both the connection to the internal network and the internet is always on to the update server. This way, the update server can continuously update its repository and distribute the update to internal computers. The update server acts similar to a router that allow the computers in the internal network to access the internet, except that the only kind of data that can get through is updates and this data also subjects to approval in WSUS.

This model requires an additional interface for the update server. However, the server can also perform other roles for the internal network, since it is always connected.

This model is the least secure of all three. It poses a serious security threat to the internal network in case the update server is compromised by a malicious party. Therefore, making sure the update server itself is up-to-date with the latest security patches and strict firewall configuration is a must.

Comparison between models

Two servers One server, switch One server, always on
Security ranking High Medium Low
Hardware cost Expensive Minimal Medium
Configuring cost Medium (Two servers and transfer media setup) Low (One server and firewall setup) Low
Management cost High (Update approval on internal server and decide when to sync.) Medium (Need to decide when to switch) Low (Update approval only)
Maintenance cost High (Manual sync.) Medium (Manual/Automatic switch required) Low (Usual security and maintenance checks only)

Configuring WSUS servers

WSUS setup process is straightforward. A configuration wizard is started after WSUS installation is completed to help users configure WSUS. This wizard can be accessed later through the option item in WSUS configuration snap-in (Control panel à Administrative tools à Microsoft Windows Server Update Services)

image008

Figure 6 Options item

WSUS can be configured to pull data from another WSUS server instead of the default Microsoft server (useful when deploying the two server model, connected with an ad-hoc network). Other configuration options are visible on Figure 6 and to the left of Figure 7. It should be noted that this wizard should not be followed on the internal server in the two server model since it requires a direct connection to the upstream update server to continue past step 4. Actually no configuration is needed in this case.

image8

Figure 7 Update source configuration screen

Managing approvals and installing updates

Computers in the internal network could be configured into groups to manage which update should or should not be installed on them. WSUS can only manage clients which have reported to it. Refer to section “Configuring clients” for details on how to configure clients.

image9

Figure 8 Client management screen, listing all active clients

image10

Figure 9 Update approval / rejection

image11

Figure 10 Detailed status report for a client, with report viewer installed

After being properly configured, clients will automatically pull updates from the WSUS server and install them if those update packages are approved and have not been installed already.

Synchronizing update repositories

Deploying the two servers model with tapes and external HDDs require manual export and import of update data. “Update data” includes:

  • Package executables themselves, which are saved as files in WSUS’ update repository (the default folder is C:WSUSWsusContent and can be changed during WSUS’ setup). This is synchronized simply by copying the content of the entire folder.
  • Information about the packages such as checksum and verification data, termed “metadata”. See “Exporting metadata” and “Importing metadata” for details.

WSUS requires both kind of data to be synchronized to work.

Usually, the list of available updates and their metadata is downloaded before the update packages themselves. WSUS will not distribute such incomplete packages.

image12

Figure 11 Warning when package’s files have not been downloaded

Exporting metadata

Syntax: wsusutil export <datafile> <logfile>. This will produce a metadata container file (.cab) and a log file (.log). Both are needed for metadata import. The exporting process could take a while.

image13

Figure 12 Exporting metadata

Importing metadata

Syntax: wsusutil import <datafile> <logfile>.

image14

Figure 13 Metadata import

Configuring clients

To make client update through the new WSUS server instead of the default (Microsoft’s server), the policy for the domain should be modified as in Figure 14 to point to the new WSUS server: http://<servername>.

image15

Figure 14 Client configuration (illustrated with local policy editor)

To edit corporate policy, you con either use the Group Policy Management Console, right click the OU you want to apply WSUS policy and select create policy, you should now see he Group policy object editor similap to tho above screenshot.

Alternatively, you can open Active Directory users and computers, open the properties windows for the OU you want, select the policy tab and click Add (or edit an existing policy).

After policy modification, it may be necessary to:

  • Forces refresh the active policy with gpupdate /force (which will be automatically refreshed in 15 minutes anyways).
  • Clear Internet Explorer’s cache on the client to force reload of update configuration files downloaded from the previous update server.
  • Reauthorize Automatic updates on the client so it is registered on the update server with wuauclt.exe /resetauthorization /detectnow.
  • Make sure that Automatic Update for the client is enabled.

image16

Figure 15 Windows XP Automatic updates configuration screen (My computer à Properties)

Upon successful configuration, the client will appear on WSUS configuration snap-in as in Figure 8.

Downloads

1. WSUS 3.0 SP2: Main WSUS Component.

2. Update fix for WSUS 3.0 SP1: WSUS 3.0 SP1 is known to have a bug that make some client unable to download updates.

3. Microsoft report viewer 2005: Required to view detailed reports.

4. Windows Installer 3.1: WSUS perquisite.

5. .net Framework 2.0 SP1: WSUS perquisite.

6. Background intelligent transfer service 2.0 update: WSUS perquisite.

7. Windows Server 2003 SP2: WSUS perquisite.

References

1. Microsoft Windows Server Update Services 3.0 SP2 Deployment Guide

2. Microsoft Windows Server Update Services 3.0 SP2 Operations Guide

Disclaimer

Registered trademarks are property of respective owners. This document contains no sensitive or specific information about any corporate entity.

Report @ localhost

Thanh Phu on May 7, 2009

I was developing an automated report maker. Its mission is to collect data from various sources (MSSQL, Excel and Access) to create a summary of the numbers. Things hit a few bumps along the way but finally after 2 days, the collecting phase (the one I thought was hard) is complete.

On to the “easy” part…

C# is such a great environment that they could have offered you with two report solution: the standard report control that comes with the framework and Business Object’s Crystal Report. I was looking for something that could accept parameters, use that for some SQL queries and display the result with a couple groupings.

Crystal report as overtly complex for the purpose, heh, and adding parameter to the query is not that easy as it depends on the project’s datasource.

The report control is even worse. I vividly recall from the last time I worked with such reports they have three tabs in the designer for me to edit data, layout and preview the result. But to my surprise, I can’t even find the preview command anywhere after I created a report in a Winform application. Not on the toolbar, not in the menu, nowhere to be found!

After a little twiddling around, I found another kind of project dedicated to report: the report wizard from the Business Intelligence studio (I happen to have SQL Server Express installed on the development machine). Exactly what I’m looking for! I just paste in the query, choose the groupings and voilà! Report to print! And they have three tabs…

I thought to myself it was all Microsoft stuff, and maybe the report viewer back from the winform project could view this, so I copied it over, point the control to the report, and execute it.

“Data source not found” is all I get.

It turned out that the world isn’t all pink like Microsoft thought (see the “just rename your file and it’ll work, we are Microsoft!” article). The business intelligence’s report is to be and only to be deployed on a report server, which is really not convenience for me since my version of SQL is Express and does not include the report server :(. From a winform application, the report viewer control will just ignore the queries you spent hours crafted in the .rdl report when you rename it to .rdlc (Because they name it report for different meanings on different tiers – A total solution on the server and a mere viewer on the client). They say you have to rebind the datasource. Easy for them to say when they aren’t binding a lot of queries together – by hand since Microsoft’s great data designer won’t allow you to create a new table with schema based on your queries, the situation is even worse for parameterized queries! =__=

Queries are naughty, so no rows for you!

Queries are naughty, so no rows for you!

I'm too lazy to parse that text (the same query run just fine from SSMS)

I'm too lazy to parse that text (the same query run just fine from SSMS)

Folks on the ‘net have not been helpful this time. All they did was point back to the article, It’s surprising to see that something not impossible as viewing .rdl locally is impossible to do with what Microsoft allow you to use. (Yes, they allow you to preview the report locally just fine but they hid the damn report viewer!)

Who needs Microsoft when we have the Apache license? 😛

Somebody did it the hard way: they created a new parser and render the report from scratch. It have evolved for sometime: 0.5 from the like above to 4.0 latest. But still I does have limitations: some attributes are yet to be parsed and are ignored (for example, tables will not expand to accomodate the records) so the report may look a little off, but that could be fixed with a simple moving of parts. After all, this is open source software: if you wanted it to be better, dig in! 🙂

Anyways, it gets the job done. My reports are displaying expected numbers and passing parameters are painless. Cheers to the author!

RDL reports being viewed locally

RDL reports being viewed locally

Latest version of project RDL is available at FYIReporting’s homepage.

UPDATE: And don’t think you can just bind your data with the dataset designer, it won’t tell you anything when you choose the data set as the data source, but when you run it you get an ugly “not bind to an instance” error. In the end you’ll have to do things manually with Microsoft.
reportViewer1.LocalReport.DataSources.Clear();
reportViewer1.LocalReport.DataSources.Add(
                new ReportDataSource("Table name", new DataSetName.TableAdapter().GetData(Parameters))
);