How to use Linux hw-probe behind a HTTP Proxy (for probe upload to linux-hardware.org)

Written by - 0 comments

Published on - last updated on June 15th 2021 - Listed in Linux Perl Coding Hardware


Behind the public hardware database linux-hardware.org is basically one open source program: hw-probe. This little helper tool collects information from local hardware devices, tries to determine whether or not they are correctly working and uploads the "probe" into the database of linux-hardware.org.

This database is a great helper if you want to buy new hardware but Linux compatibility is not (well) documented of that hardware device. I've used this database to check Linux compatibility in Spring 2021 when my main workstation for daily work received a hardware upgrade (ASRock B550 and AMD Ryzen 5 3600 on Linux Mint 20.1 with Kernel 5.8).

Probe upload fails

To upload the probe, an Internet connection is obviously required. However there are many environments, where direct Internet access is not possible or blocked by a firewall. hw-probe then fails to upload the probe:

root@linux:~# sudo -E hw-probe -all -upload
Probe for hardware ... Ok
Reading logs ... WARNING: X11-related logs are not collected (try to run 'xhost +local:' to enable access or run as root by su)
Ok
read failed: Connection reset by peer at /usr/share/perl5/LWP/Protocol/http.pm line 382.

ERROR: failed to upload data

In such environments, an outgoing http/https proxy is often used. But even by defining the proxy using environment variables, the probe upload would still fail:

root@linux:~# export https_proxy=http://myproxy.example.com:3128
root@linux:~# export http_proxy=http://myproxy.example.com:3128
root@linux:~# sudo -E hw-probe -all -upload
Probe for hardware ... Ok
Reading logs ... WARNING: X11-related logs are not collected (try to run 'xhost +local:' to enable access or run as root by su)
Ok
read failed: Connection reset by peer at /usr/share/perl5/LWP/Protocol/http.pm line 382.

ERROR: failed to upload data

The errors reveal that in the background the Perl module LWP is used.

Analyzing the source code

As /usr/bin/hw-probe is a Perl script, the source code can be read and modified (without having to run a ton of compilers). The relevant upload part indeed starts a new LWP::UserAgent connection in the script's postRequest function:

sub postRequest($$$)
{
    my ($UploadURL, $Data, $SSL) = @_;

    require LWP::UserAgent;

    my $UAgent = LWP::UserAgent->new(parse_head => 0);

    if($SSL eq "NoSSL" or not checkModule("Mozilla/CA.pm"))
    {
        $UploadURL=~s/\Ahttps:/http:/g;
        $UAgent->ssl_opts(verify_hostname => 0, SSL_verify_mode => 0x00);
    }

    my $Res = $UAgent->post(
        $UploadURL,
        Content_Type => "form-data",
        Content => $Data
    );

    my $Out = $Res->{"_content"};

    if(not $Out) {
        return $Res->{"_headers"}{"x-died"};
    }

    return $Out;
}

The LWP::UserAgent module also supports a proxy as option, but obviously this is not implemented here.

Adding a proxy option to hw-probe

To add support for outgoing http/https proxies, I modified the postRequest function, checking whether or not the command line parameter -proxy (which is new) was used and then use the UserAgent option 'proxy':

sub postRequest($$$)
{
    my ($UploadURL, $Data, $SSL) = @_;

    require LWP::UserAgent;

    my $UAgent = LWP::UserAgent->new(parse_head => 0);

    if($Opt{"Proxy"}) {
        my $proxy = $Opt{"Proxy"};
        $UAgent->proxy([ [ 'http', 'https' ] => "http://$proxy" ]);
    }


    if($SSL eq "NoSSL" or not checkModule("Mozilla/CA.pm"))
    {
        $UploadURL=~s/\Ahttps:/http:/g;
        $UAgent->ssl_opts(verify_hostname => 0, SSL_verify_mode => 0x00);
    }

    my $Res = $UAgent->post(
        $UploadURL,
        Content_Type => "form-data",
        Content => $Data
    );

    my $Out = $Res->{"_content"};

    if(not $Out) {
        return $Res->{"_headers"}{"x-died"};
    }

    return $Out;
}

To see the full diff:

$ git diff
diff --git a/hw-probe.pl b/hw-probe.pl
index 3d1a084..8a2b0a2 100644
--- a/hw-probe.pl
+++ b/hw-probe.pl
@@ -167,6 +167,7 @@ GetOptions("h|help!" => \$Opt{"Help"},
   "upload|confirm-upload-of-hashed-ids!" => \$Opt{"Upload"},
   "hwinfo-path=s" => \$Opt{"HWInfoPath"},
   "log!" => \$Opt{"ShowLog"},
+  "proxy=s" => \$Opt{"Proxy"},
 # Inventory
   "inventory|inventory-id|i|group|g=s" => \$Opt{"Group"},
   "generate-inventory|generate-inventory-id|get-inventory-id|get-group!" => \$Opt{"GenerateGroup"},
@@ -355,6 +356,9 @@ GENERAL OPTIONS:
   -hwinfo-path PATH
       Path to a local hwinfo binary.

+  -proxy address:port
+      Set outgoing http/https proxy using syntax: proxy.domain.local:3128
+
 INVENTORY OPTIONS:
   -i|-inventory-id ID
       Mark the probe by inventory ID.
@@ -2475,6 +2479,11 @@ sub postRequest($$$)
     require LWP::UserAgent;

     my $UAgent = LWP::UserAgent->new(parse_head => 0);
+
+    if($Opt{"Proxy"}) {
+        my $proxy = $Opt{"Proxy"};
+        $UAgent->proxy([ [ 'http', 'https' ] => "http://$proxy" ]);
+    }

     if($SSL eq "NoSSL" or not checkModule("Mozilla/CA.pm"))
     {

The changes were pushed to the upstream project as Pull Request #102.

Running hw-probe with a proxy

Running hw-probe now with the new -proxy parameter and the proxy defined in the command line:

root@linux:~# sudo -E hw-probe -all -upload -proxy myproxy.example.com:3128
Probe for hardware ... Ok
Reading logs ... WARNING: X11-related logs are not collected (try to run 'xhost +local:' to enable access or run as root by su)
Ok
Uploaded to DB, Thank you!

Probe URL: https://linux-hardware.org/?probe=a35e7d3eb0

And the probe was sucessfully uploaded!

Open Source Rulez!

Update: The code has been merged

The code to add the possibility of using an outgoing proxy has been merged into the upstream project by the hw-probe maintainer.


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder