Webdav cadaver: script possible?

[Richard Ivarson]

Hi,

with the ugly sounding but nice Cygwin tool "cadaver" (a command line Webdav
client, like ftp) I would like to automatically download some files from the
Webdav server to my local drive.

However, in contrast to for example lftp, I don't see a way how to do the few
commands automatically.

cadaver https://my_webdav.site.com

this asks for username and password, then enters cadaver's command mode

there I've to enter:
* mget the_files
* quit

In case there's no way to use a script, would it be possible to "somehow"
give all the command lines to the console which runs the cadaver program?

Thanks.

-ric

[Richard Ivarson]

... or maybe there is another client/programm to access Webdav from within
Cygwin?


... or is there any way to feed scripted lines to a standard Cygwin command
line programm? (Re-direction, somehow?)

-ric

[Thorsten Kampe]

* Richard Ivarson (Mon, 10 Sep 2007 21:37:42 +0200)

Post by Richard Ivarson
... or maybe there is another client/programm to access Webdav from within
Cygwin?
... or is there any way to feed scripted lines to a standard Cygwin command
line programm? (Re-direction, somehow?)

Expect?

[Richard Ivarson]

Post by Thorsten Kampe
* Richard Ivarson (Mon, 10 Sep 2007 21:37:42 +0200)

Post by Richard Ivarson
... or maybe there is another client/programm to access Webdav from within
Cygwin?
... or is there any way to feed scripted lines to a standard Cygwin command
line programm? (Re-direction, somehow?)

Expect?

Thanks for that hint, Thorsten.
I didn't know "expect" but it seems very powerful, useful -- and complex to
master. :-)

Meanwhile due to your hint I've remembered the "simple" redirection method in
Windows (and similar in Unix), and tried:

cadaver < lines.txt

with "lines.txt" holding these three lines:
USERNAME
PASSWORD
ls
<<

It partly works: I see that it connects to the Webdav server and prints the
directory, so it must have been logged in correctly.
However somehow it never comes to line three but just returns...
I've got no idea why and also I've got no plan how to make "except" doing the
little task.

-ric

[Stepp, Charles]

I'm a unix sysadmin/oracle DBA. For me, Expect is the killer app of
Cygwin, even over the X server. Get the O'Reilly Expect book by Don
Libes. It is a treasure trove of Unix knowledge as well as Expcect/TCL.
Check out the autoexpect script. You may have to find it somewhere to
install it into the Cygwin distro, but it records your keystrokes and
creates a primitive expect script to redo the steps. You can take the
basic script and edit it to ensure it is efficient and robust. Expect
scripting requires that you understand the interactive interaction you
are trying to automate and it requires that you learn some of the magic
of globs and regular expressions. There are some great debug flags that
will show you exactly what the interpreter is thinking. Great stuff.

-----Original Message-----
From: Richard Ivarson [mailto:***@gmail.com]
Sent: Tuesday, September 11, 2007 4:48 PM
To: ***@cygwin.com
Subject: Re: Webdav cadaver: script possible?

Post by Thorsten Kampe
* Richard Ivarson (Mon, 10 Sep 2007 21:37:42 +0200)

Post by Richard Ivarson
... or maybe there is another client/programm to access Webdav from within
Cygwin?
... or is there any way to feed scripted lines to a standard Cygwin command
line programm? (Re-direction, somehow?)

Expect?

Thanks for that hint, Thorsten.
I didn't know "expect" but it seems very powerful, useful -- and complex
to
master. :-)

Meanwhile due to your hint I've remembered the "simple" redirection
method in
Windows (and similar in Unix), and tried:

cadaver < lines.txt

with "lines.txt" holding these three lines:
USERNAME
PASSWORD
ls
<<

It partly works: I see that it connects to the Webdav server and prints
the
directory, so it must have been logged in correctly.
However somehow it never comes to line three but just returns...
I've got no idea why and also I've got no plan how to make "except"
doing the
little task.

-ric

[Gregg Reynolds]

Post by Richard Ivarson

cadaver https://my_webdav.site.com

this asks for username and password, then enters cadaver's command mode
* mget the_files
* quit

If all you're only doing standard http methods try wget. curl if you
need real webdav methods.

[Richard Ivarson]

Post by Gregg Reynolds

Post by Richard Ivarson

cadaver https://my_webdav.site.com

this asks for username and password, then enters cadaver's command mode
* mget the_files
* quit

If all you're only doing standard http methods try wget. curl if you
need real webdav methods.

Hi Gregg,

thanks for this good hint!

Indeed I'd just like to fetch, say, five files from my Webdav server
automatically with a script (which I start however and so can enter one
password at its begin, for example)

A) *wget*
Fetching the files with wget works:
wget --user=NAME --password=PASS https://webdavserver.com/folder/fileAA.txt

I've seen two handicaps:
1) I want to use a script so I don't want to store the password in the script
file. Ommiting "--password" doesn't ask for it, however.
2) Not a big one because I could specify them: I'd like to be able to get
fileBB ... fileEE.txt too.


B) *curl*
You mentioned "curl" which I didn't know yet but I do now (a bit) ! It
actually does what I've been looking for. So no need to run a complex
"expect" command on "cadaver".

curl --user NAME:PASSWORD
https://webdavserver.com/folder/{fileAA,fileBB,fileCC,fileDD,fileEE,}.txt -O
-O -O -O -O


Just great. How I love these Unix tools - and Cygwin which makes Windows
useable. And of course the people helping each other.

Btw, I still got two small questions:
1) Is there a simpler way than the five "-O"'s I used? Would a wildcard be
possible (didn't find a mention in the manpage, however).
2) What actually is the main difference between wget and curl ?

Again thanks and all the best.

-ric

[Richard Ivarson]

Richard Ivarson schrieb:
[..]

Post by Richard Ivarson
A) *wget*
wget --user=NAME --password=PASS
https://webdavserver.com/folder/fileAA.txt
1) I want to use a script so I don't want to store the password in the script

[..]

Post by Richard Ivarson
curl --user NAME:PASSWORD
https://webdavserver.com/folder/{fileAA,fileBB,fileCC,fileDD,fileEE,}.txt
-O -O -O -O -O

... I forgot to mention: In contrast to A)1) with curl you can ommit the
":PASSWORD" in the script. The curl then nicely asks for it at runtime.


-ric

[Steve Holden]

Post by Richard Ivarson
[..]

Post by Richard Ivarson
A) *wget*
wget --user=NAME --password=PASS
https://webdavserver.com/folder/fileAA.txt
1) I want to use a script so I don't want to store the password in the script

[..]

Post by Richard Ivarson
curl --user NAME:PASSWORD
https://webdavserver.com/folder/{fileAA,fileBB,fileCC,fileDD,fileEE,}.txt
-O -O -O -O -O

... I forgot to mention: In contrast to A)1) with curl you can ommit the
":PASSWORD" in the script. The curl then nicely asks for it at runtime.

I don't know about getting rid of the five -O's (have you tried it with
just one to see whether that does what you want?), but you could shorten
the command line a bit using

https://webdavserver.com/folder/file{AA,BB,CC,DD,EE}.txt

The shell quite happily expands braces as a portion of a filename:

***@bigboy ~
$ echo https://server/path/file{AA,BB,CC,DD,EE}.txt
https://server/path/fileAA.txt https://server/path/fileBB.txt
https://server/path/fileCC.txt https://server/path/fileDD.txt
https://server/path/fileEE.txt

regards
Steve

[Richard Ivarson]

Steve Holden schrieb:
[..]

Post by Steve Holden

Post by Richard Ivarson

Post by Richard Ivarson
curl --user NAME:PASSWORD
https://webdavserver.com/folder/{fileAA,fileBB,fileCC,fileDD,fileEE,}.txt
-O -O -O -O -O

... I forgot to mention: In contrast to A)1) with curl you can ommit
the ":PASSWORD" in the script. The curl then nicely asks for it at
runtime.

I don't know about getting rid of the five -O's (have you tried it with
just one to see whether that does what you want?),

Yes, I tried: with one "-O" just the first "fileAA.txt" will be written to
local file ("fileAA.txt"), while the other files will be printed to the
standard output (console).

Post by Steve Holden
but you could shorten
the command line a bit using
https://webdavserver.com/folder/file{AA,BB,CC,DD,EE}.txt

Nice.

Best wishes.

-ric

[Buchbinder, Barry (NIH/NIAID) [E]]

Post by Richard Ivarson
[..]

Post by Steve Holden

Post by Richard Ivarson
curl --user NAME:PASSWORD
https://webdavserver.com/folder/{fileAA,fileBB,fileCC,fileDD,fileEE,
}.txt -O -O -O -O -O

... I forgot to mention: In contrast to A)1) with curl you can omit
the ":PASSWORD" in the script. The curl then nicely asks for it at
runtime.

I don't know about getting rid of the five -O's (have you tried it
with just one to see whether that does what you want?),

Yes, I tried: with one "-O" just the first "fileAA.txt" will be
written to local file ("fileAA.txt"), while the other files will be
printed to the standard output (console).

Post by Steve Holden
but you could shorten
the command line a bit using
https://webdavserver.com/folder/file{AA,BB,CC,DD,EE}.txt

Nice.

Curl will also do it by itself. (You may have to quote the URL so that the shell does not expand the it.) The following is from the man page. Note the {curly braces}.

URL
The URL syntax is protocol dependent. You'll find a detailed
description in RFC 3986.

You can specify multiple URLs or parts of URLs by writing
part sets within braces as in:

http://site.{one,two,three}.com

or you can get sequences of alphanumeric series by using [] as in:

ftp://ftp.numericals.com/file[1-100].txt
ftp://ftp.numericals.com/file[001-100].txt (with leading zeros)
ftp://ftp.letters.com/file[a-z].txt

No nesting of the sequences is supported at the moment, but you
can use several ones next to each other:

http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html

[Matthew Woehlke]

Post by Richard Ivarson
1) I want to use a script so I don't want to store the password in the
script file. Ommiting "--password" doesn't ask for it, however.

'ask for password' is planned in a future release of wget (probably the
next-to-next release).

[Brian Dessent]

Post by Richard Ivarson
Just great. How I love these Unix tools - and Cygwin which makes Windows
useable. And of course the people helping each other.
1) Is there a simpler way than the five "-O"'s I used? Would a wildcard be
possible (didn't find a mention in the manpage, however).

You need to provide one -O for each URL. I don't think there's a way
around that. If you use -o instead of -O you can use "#n" as a
replacement variable to represent the stuff that was in the n-th {} or
[]. Try

curl --user NAME:PASSWORD
'https://webdavserver.com/folder/file{AA,BB,CC,DD,EE}.txt' -o
"file#1.txt"

Here the URL is single quoted so that curl does the expansion of {} and
not the shell, that way it is able to expand #1 into the contents of the
first {} operator.

Post by Richard Ivarson
2) What actually is the main difference between wget and curl ?

They are different projects with different goals. There is a comparison
chart at: <http://curl.haxx.se/docs/comparison-table.html>.

curl the command line tool is actually just a thin wrapper around
libcurl the library, which is a general purpose library that can be used
in any project. This sets it apart from wget as it makes it very easy
to add http/ftp/et at. client capabilities to an application. The fact
that it's BSD licensed also facilitates this, as can be seen at:
<http://curl.haxx.se/docs/companies.html>.

In my experience wget is more suited for doing things like downloading a
recursive copy of an entire site including images and stylesheets. It
even has an option for converting the locally downloaded copy to have
links to the local copies of these items (instead of absolute http:
URLs) so that it can be viewed offline.

Curl on the other hand is better suited for http scripting where you are
trying to emulate the actions of a user, such as when submitting form
fields. Wget has the ability to do http POSTs but it only supports the
old "application/x-www-form-urlencoded" content encoding and not the
more sophisticated "multipart/form-data" encoding which allows for
things like file uploads. Curl supports both.

Brian

[Gregg Reynolds]

Post by Brian Dessent
Curl on the other hand is better suited for http scripting where you are
trying to emulate the actions of a user, such as when submitting form
fields. Wget has the ability to do http POSTs but it only supports the
old "application/x-www-form-urlencoded" content encoding and not the
more sophisticated "multipart/form-data" encoding which allows for
things like file uploads. Curl supports both.

Curl is also great for examining protocol transactions for any of the
supported protocols. Great tool for e.g. learning what webdav
transactions actually look like. I wouldn't want to try to debug a
networking application without it, that's for sure.

-g

[Gregg Reynolds]

Post by Richard Ivarson
curl --user NAME:PASSWORD
https://webdavserver.com/folder/{fileAA,fileBB,fileCC,fileDD,fileEE,}.txt -O
-O -O -O -O

...

Post by Richard Ivarson
1) Is there a simpler way than the five "-O"'s I used? Would a wildcard be
possible (didn't find a mention in the manpage, however).

Dunno. I actually haven't used it much; I came across it when I
wanted a tool for managing webdav resources, but ran into a problem
involving %-encoding of the space character. Never found out if it
was a problem with curl or with the server, but I found a different
method and haven't had time to go back to curl. What I really want is
a tool that can manage webdav properties in batch mode; if you know of
one please let me know. You could do it with curl, but since curl
doesn't natively understand webdav it would be a good bit of work.

For the -O issue: as with any tool that doesn't quite behave the way
you like, you can always try The Unix Way: feed the output of one
small tool into another. Download without -O, then use a shell
command or script to convert the results. Maybe something along the
lines of

$ find ./ -name "*.txt" -exec basename {} \;.

You might even be able to pipe the curl command output into basename.

-gregg

[Richard Ivarson]

Many thanks to all posters on the thread; your comments have been very
helpful and it works fine now.
Also I learned the differences between wget and curl, too.

All the best!

-ric

[Jeff Hawk]

-----Original Message-----
From: On Behalf Of Richard Ivarson
Sent: Wednesday, September 12, 2007 9:42 AM
Subject: Re: Webdav batch file transfer: curl, wget
[..]

curl --user NAME:PASSWORD

https://webdavserver.com/folder/{fileAA,fileBB,fileCC,fileDD,fileEE,}.
t
xt

Post by Steve Holden

Post by Richard Ivarson

Post by Richard Ivarson
-O -O -O -O -O

... I forgot to mention: In contrast to

A)1) with curl you can

Post by Steve Holden

Post by Richard Ivarson
ommit the ":PASSWORD" in the

script. The curl then nicely asks for

Post by Steve Holden

Post by Richard Ivarson
it at runtime.

I don't know about getting rid of the five -O's

(have you tried it

with

Post by Steve Holden
just one to see whether that does what you

want?),

Yes, I tried: with one "-O" just the first

"fileAA.txt" will be

written to local file ("fileAA.txt"),

while the other files will be

printed to the standard output (console).

Post by Steve Holden
but you

could shorten

Post by Steve Holden
the command line a bit using
https://webdavserver.com/folder/file{AA,BB,CC,DD,EE}.txt

Try quoting the URL.
curl --user NAME:PASSWORD "https://webdavserver.com/folder/file{AA,BB,CC,DD,EE}.txt"
-O





If you are using bash, it is splitting the file names
into 5 separate names on the command line before curl gets them.


Then a single -O should work fine.





Regards,


Jeff