(WARNING - under development as in everything is probably broken!)
Curlpipe is a little DSL making it easy to build up http execution pipelines.
Curlpipe uses libcurl under the covers, curl is a great swiss army knife http client that does much more then just make it easy to work with HTTP, supporting a plethora of URI addressable protocols. Over the years, the curl command line interface has grown - exposing many options with most users only ever invoking a subset of features, learning more advanced features over time.
Getting Started | Usage | Language | Examples | Points of Interest | Developing | License
Eventually you will be able to Download the latest release for your platform but today you are on the bleeding edge and must build the software.
To try it out,
echo "[http://www.httpbin.org/get] > [/tmp/output.txt]" | curlpipeor define a file (example.cp)
[http://www.httpbin.org/get] > [/tmp/output.txt]and invoke curlpipe, supplying that file as its only argument.
> curlpipe example.cpYou should now observe the output from the URI saved to a file.
It is easy to send data eg.
"name=Jasmine&age=1" | [http://www.httpbin.org/post]Some more examples.
To get help run
> curlpipe -hcurlpipe 0.1.0 | â“’ 2017-2018 James Fuller <[email protected]> | https://github.com/xquery/curlpipe
> curlpipe mycurlpipe.cp
-h | --help : Help.
-d | --debug : Emit debug info logging.
-i | --info : Emit info logging.
-l | --log : Enable logging to file.
-q | --quiet : Suppress output to stdout (console).
-a | --auth : Pass a username:password pair as the argument.
-A | --auth-type : Specify the auth mechanism (basic|digest).
-p | --params : Define set of parameters for token replacement (json|xml).
-P | --param : Define parameter(s) for token replacement.
-o | --options : Define set of options controlling curlpipe behavior (default is ~/.curlpiperc).
-O | --option : Define option(s) controlling curlpipe behavior.Curlpipe flags for controlling how much information is emitted during processing.
| short | long | description |
|---|---|---|
| -h | --help | Obtain help |
| -d | --debug | Emit debug info logging. |
| -i | --info | Emit info logging. |
| -l | --log | Enable logging to file. |
| -q | --quiet | Suppress output to stdout (console). |
The following flags set the default auth credentials and auth-type used by curlpipe.
| short | long | description |
|---|---|---|
| -a | --auth | Pass a username:password pair as the argument. |
| -A | --auth-type | Specify the auth mechanism (basic |
~/.netrc (on windows ~/_netrc) is also supported by curlpipe (by dint of libcurl support):
> cat ~/.netrc
machine example.org
login myuser
password mypasswordParameters maybe passed to curlpipe, replacing tokens in the curlpipe script.
| short | long | description |
|---|---|---|
| -p | --params | set file containing params (xml |
| -P | --param | set a param (ex. -Pid=1). |
Parameters can be passed in via a file, for example, using data.json:
{"id":1, "name":"Ali G"}The call to curlpipe would be
> curlpipe -p data.json example.cpor data.xml looks like
<params>
<param>
<name>id</name>
<value>1</value>
</param>
<param>
<name>name</name>
<value>Ali G</value>
</param>
</params>and similarly called
> curlpipe -p data.xml example.cpParamaters may also be individually defined, overidding params if used in conjunction with above -p flag.
> curlpipe -Pid=1 -Pname=Ali G example.cpBy default, curlpipe looks for a ~/.curlpiperc file (on windows _curlpiperc) containing options that control curlpipe processing behavior. An example ~/.curlpiperc file is here
| short | long | description |
|---|---|---|
| -o | --options | set file containing curlpipe options (xml |
| -O | --option | set an option (ex. -Pid=1). |
Alternately you can set the location of this file with the -o flag.
Options can be overriden at the command line using the -O flag. The set of options curlpipe exposes are:
| option | example | description |
|---|---|---|
| TBA. | ||
| TBA. |
Curlpipe defines a series of statement(s). The simplest statement defines retrieval of a URI:
[http://www.httpbin.org/get] ;where the seperator (;) is optional.
When this is run the output of dereferencing the URI is sent to stdout (con).
In addition to a URI, curlpipe supports boolean, literal, binary, xml, json and null data types.
| data type | example | description |
|---|---|---|
| URI | [http://www.example.org] |
used to POST, PUT or DELETE. |
| literal | "name=value;name=value" |
literal data value. |
| binary | ex. zip file | binary data value. |
| xml | <person><name>Tommy</name></person> |
well formed xml data. |
| json | {id:1,name:"Tommy"} |
json data. |
| boolean | ? ? |
true or false. |
| null | [] |
an empty/null value. |
Literal string data can be used to pass in name value pairs (using application/x-www-form-urlencoded content type):
"name=value&name=value" ;Binary data (for example, a zip file) is also supported.
Support for common formats, like XML and json.
"<person><name>Tommy</name></person>" ;"{id:1,name:'Tommy;}";A single curlpipe statement can be comprised of
(datatype | datatype operator datatype)
either a datatype or a datatype + operator + datatype where operators perform actions (or test conditions) on datatypes.
The following example illustrates how the contents of /temp/data.json is sent to a HTTP endpoint supporting POST method.
[/tmp/data.json] | [http://httpbin.org/post]If the endpoint URI supports PUT it may opt to use that. If the endpoint does not support these methods then the appropriate http error code is thrown.
"name=Jim&age=21" | [http://httpbin.org/post]The following would POST xml to the URI.
"<person><name>Tommy</name></person>" | [http://httpbin.org/post]or now with a json datatype.
"{id:1,name:'Tommy'}" | [http://httpbin.org/post]The pipe operator can be used to perform an HTTP DELETE
[] | [http://httpbin.org/delete]Like the PUT the pipe operator will deduce if the endpoint supports DELETE. This built in behavior is enabled by curlpipe options (which can be set when invoking curlpipe).
Otherwise one is always free to force a PUT
"{id:1,name:'Tommy'}" =| [http://httpbin.org/put]To force a DELETE method
[] =| [http://httpbin.org/delete]To force a HEAD method use a conditional (explained further down)
[http://httpbin.org/head] != []Operators chain together to build execution pipeline of arbitrary length.
[/tmp/data.json] | [http://httpbin.org/post] > [/tmp/output.txt] | [http://httpbin.org/post]The set of processing operators are:
| operator | description |
|---|---|
| | | used to POST, PUT or DELETE. |
| > | redirect output to file. |
| >> | append output to file. |
| =| | used to force PUT or DELETE. |
Parameters can be passed into curlpipe
> curlpipe -Pname=Tommy example.cp -Pid=1and used for token replacement (ex. ${token}) in either data or URIs.
{name:"${name}"} | [http://httpbin.org/get/${id}]curlpipe implements boolean and null datatype which can be used with conditional operators to test data values.
[http://www.httpbin.org/get] == "test"[http://www.httpbin.org/get] ~= "test"Where conditionals can be composited up using AND(&&) or OR(||) operators.
[http://www.httpbin.org/get] ~= "test" && [http://www.httpbin.org/get] != "not test"curlpipe supports boolean logic, in the following form.
[http://www.httpbin.org/get] =~ "test" > [/tmp/matches.txt]where the file is only written if the match is a success.
Additionally, curlpipe supports trinary logic, in the following form.
[http://www.httpbin.org/get] =~ "test"
? > [/tmp/success.txt]
: 2> [/tmp/fail.txt]The set of conditional operators are:
| operator | description |
|---|---|
| == | equal |
| != | does not equal |
| ~= | regex text |
| && | AND chain condition |
| || | OR chain condition |
You can narrow down data using selectors, with postfix defining a simple xpath like selection.
[http://localhost:81/image/svg].svg.title | [http://localhost:81/post "Content-type":"application/xml"]Where the above example will retrieve elements under svg/title ... this simple path selection works equally across xml or json.
Curlpipe defines the following options
| option | type | default | description |
|---|---|---|---|
| deduce-methods-on-pipe | boolean | true | will detect if URI supports PUT or DELETE when using pipe operator. |
Find more examples here.
[http://www.httpbin.org/get] > [/tmp/output.txt][http://www.httpbin.org/get auth=myser:password auth-type=digest] > [/tmp/output3.txt][http://www.httpbin.org/get "Accept":"application/json"][http://www.httpbin.org/get] >> [/tmp/response.txt][http://www.httpbin.org/get],[http://www.httpbin.org/uuid] >> [/tmp/response.txt][http://www.httpbin.org/image/svg].svg.title | [http://www.httpbin.org "Content-type":"application/xml"][http://www.httpbin.org/get] =~ "test" > [/tmp/matches.txt][http://www.httpbin.org/get] =~ "test"
? > [/tmp/success.txt]
: > [/tmp/fail/fail.txt]
"{'id':1 , "name":'James Fuller' age='${age}'}" | [http://www.httpbin.org/post] ;[/tmp/data.json] | [http://www.httpbin.org/post "Content-type":"application/json"] ;"<person id="1"><name>John Smith<name><age>${age}</age></person>" | [http://www.httpbin.org/post] ;[/tmp/data.xml] | [http://www.httpbin.org/post "Content-type":"application/xml"] ;"id=1&name=James Fuller" | [http://www.httpbin.org/post] ;[/tmp/mydoc.zip] | [http://www.httpbin.org/post "Content-type":"application/zip"] ;will deduce if endpoint supports PUT
"{"test":1}" | [http://www.httpbin.org/put]force an HTTP PUT
"{"test":1}" =| [http://www.httpbin.org/put]will deduce if endpoint supports DELETE
[] | [http://www.httpbin.org/delete]force an HTTP DELETE
[/dev/null] =| [http://www.httpbin.org/delete]"${mypayload}" | [http://www.httpbin.org/delete/${myid}]
{"id":${myid}} | [http://www.httpbin.org/delete/${myid}]
[http://www.httpbin.org/get].url == 'http://www.httpbin.org/get'[http://www.httpbin.org/get] .url != 'http://www.example.com'curlpipe is by design a 'little language' and most likely missing features from your 'favourite' language. It is intended as an adjunct to your existing script processing or host language.
To provide a framework for design thoughts, here are a few possibly non obvious gaps in the current codebase.
- curlpipe is not intended as a drop in replacement for the curl tool (which is already a great CLI).
- Intentionally lazy defining internals or worrying too much about performance at this stage.
- curlpipe language is defined with an EBNF which is used to produce a strict parser.
- Designing a programming language is hard - coherence and an easy to run AST are the first goals, please do raise an issue if you feel strongly where syntax could change.
- I find using CMake non intuitive ... its enforced usage on this project is an attempt to learn more (otherwise you would see a Makefile here!).
- Currently curlpipe is http centric in initial releases.
- Woefully ignorant of windows platform ... looking at appveyor to eventually help solve that (issue #2).
I have blatantly stolen (and deformed) concepts from many places (bash, unix pipes, prolog, etc..). The following projects provide alternate approaches to solving similar problems that curlpipe is trying to address and worth a mention.
Please raise an issue or make a contribution by forking the repository and creating a pr.
Development work happens on the [develop branch])(https://github.com/xquery/curlpipe/tree/develop) and releases are based on master branch.
To build software, run cmake:
> mkdir build
> cd build
> cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_MANUAL=OFF -DBUILD_TESTING=OFF ..
> make
> make test
> make installRunning cmake requires access to the internet to pull down dependencies. This is problematic approach for those wishing to build on a standalone machine off the network (tracked as issue 1)
The build should complain if any other dependencies are missing (ex. openssl). Please review third party dependencies for any other build requirements.
Tests require httpbin - to setup run docker-compose from top level dir (which will setup both http/https)
> docker-compose -d up
alternately, run the docker command.
> docker run -p 81:80 kennethreitz/httpbinAfter building usually it is just a matter of running make test target.
> make testor you could run directly
> cd test
> ./runAllTestslib/curlpipe/csparser.cpp is generated using REx Parser Generator with the following flags.
-name csparser -tree -cpp -fasterRelease packages are built using CPack.
Make a release build.
> mkdir build
> cd build
> cmake -DCMAKE_BUILD_TYPE=Release -DENABLE_MANUAL=OFF -DBUILD_TESTING=OFF ..
> makeThen run cpack.
> cpack --config CPackConfig.cmake
This project depends on the following external libs:
- curl: libcurl (of course!).
- REx Parser Generator: Gunther Rademacher [email protected] excellant parser generator.
- loguru: no fuss logging.
- cxxopt: parsing command args.
- googletest: testing.
- rapidjson: json munging.
- pugixml: xml dancing.
Please review these individual projects for more details on their own dependencies.
curlpipe is provided under the MIT License
MIT License
Copyright (c) 2017-2018 James Fuller [email protected]
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Getting Started | Usage | Language | Examples | Points of Interest | Developing | License