Merge pull request 'Fix disk size inventory' (#310) from ribetm/disk_size into main

Reviewed-on: #310

.envrc

@@ -0,0 +1 @@
+use nix

.gitignore

@@ -181,3 +181,6 @@ dmypy.json
 # End of https://www.gitignore.io/api/emacs,python
 
 netbox-docker
+/.vscode
+.direnv
+.pre-commit-config.yaml

MANIFEST.in

@@ -0,0 +1 @@
+include requirements.txt

README.md

@@ -13,14 +13,15 @@ The goal is to generate an existing infrastructure on Netbox and have the abilit
 * Generic ability to guess datacenters and rack location through drivers (`cmd` and `file` and custom ones)
 * Update existing `Device` and `Interface`
 * Handle blade moving (new slot, new chassis)
+* Handle blade GPU expansions
 * Automatic cabling (server's interface to switch's interface) using lldp
-* Local inventory using `Inventory Item` for CPU, RAM, RAID cards, physical disks (behind raid cards)
+* Local inventory using `Inventory Item` for CPU, GPU, RAM, RAID cards, physical disks (behind raid cards)
 * PSUs creation and power consumption reporting (based on vendor's tools)
 
 # Requirements
 
-- Netbox >= 2.6
-- Python >= 3.4
+- Netbox >= 3.7
+- Python >= 3.7
 - [pynetbox](https://github.com/digitalocean/pynetbox/)
 - [python3-netaddr](https://github.com/drkjam/netaddr)
 - [python3-netifaces](https://github.com/al45tair/netifaces)
@@ -50,7 +51,7 @@ The agent can be run from a shell and get its configuration from either the conf
 Configuration values are overridden based on the following precedence: command line arguments (might include config file) > environment variables > default config file > defaults.
 
 ```
-# netbox_agent -c /etc/netbox_agent.yml --register
+# netbox_agent -c /etc/netbox_agent.yaml --register
 INFO:root:Creating chassis blade (serial: QTFCQ574502EF)
 INFO:root:Creating blade (serial: QTFCQ574502D2) myserver on chassis QTFCQ574502EF
 INFO:root:Setting device (QTFCQ574502D2) new slot on Slot 9 (Chassis QTFCQ574502EF)..
@@ -91,10 +92,12 @@ netbox:
  token: supersecrettoken
  # uncomment to disable ssl verification
  # ssl_verify: false
+ # uncomment to use the system's CA certificates
+ # ssl_ca_certs_file: /etc/ssl/certs/ca-certificates.crt
 
 # Network configuration
 network:
-  # Regex to ignore interfaces 
+  # Regex to ignore interfaces
   ignore_interfaces: "(dummy.*|docker.*)"
   # Regex to ignore IP addresses
   ignore_ips: (127\.0\.0\..*)
@@ -110,15 +113,15 @@ network:
 # blade_role: "Blade"
 # server_role: "Server"
 # tags: server, blade, ,just a comma,delimited,list
-#
+# custom_fields: field1=value1,field2=value2#
 #
 # Can use this to set the tenant
 #
 #tenant:
 # driver: "file:/tmp/tenant"
 # regex: "(.*)"
-
-## Enable virtual machine support 
+
+## Enable virtual machine support
 # virtual:
 #   # not mandatory, can be guessed
 #   enabled: True
@@ -144,7 +147,7 @@ rack_location:
 # driver: "file:/tmp/datacenter"
 # regex: "(.*)"
 
-# Enable local inventory reporting 
+# Enable local inventory reporting
 inventory: true
 ```
 
@@ -159,6 +162,36 @@ The `get_blade_slot` method return the name of the `Device Bay`.
 
 Certain vendors don't report the blade slot in `dmidecode`, so we can use the `slot_location` regex feature of the configuration file.
 
+Some blade servers can be equipped with additional hardware using expansion blades, next to the processing blade, such as GPU expansion, or drives bay expansion. By default, the hardware from the expnasion is associated with the blade server itself, but it's possible to register the expansion as its own device using the `--expansion-as-device` command line parameter, or by setting `expansion_as_device` to `true` in the configuration file.
+
+## Drives attributes processing
+
+It is possible to process drives extended attributes such as the drive's physical or logical identifier, logical drive RAID type, size, consistency and so on.
+
+Those attributes as set as `custom_fields` in Netbox, and need to be registered properly before being able to specify them during the inventory phase.
+
+As the custom fields have to be created prior being able to register the disks extended attributes, this feature is only activated using the `--process-virtual-drives` command line parameter, or by setting `process_virtual_drives` to `true` in the configuration file.
+
+The custom fields to create as `DCIM > inventory item` `Text` are described below.
+
+```
+NAME            LABEL                      DESCRIPTION
+mount_point     Mount point                Device mount point(s)
+pd_identifier   Physical disk identifier   Physical disk identifier in the RAID controller
+vd_array        Virtual drive array        Virtual drive array the disk is member of
+vd_consistency  Virtual drive consistency  Virtual disk array consistency
+vd_device       Virtual drive device       Virtual drive system device
+vd_raid_type    Virtual drive RAID         Virtual drive array RAID type
+vd_size         Virtual drive size         Virtual drive array size
+```
+
+In the current implementation, the disks attributes ore not updated: if a disk with the correct serial number is found, it's sufficient to consider it as up to date.
+
+To force the reprocessing of the disks extended attributes, the `--force-disk-refresh` command line option can be used: it removes all existing disks to before populating them with the correct parsing. Unless this option is specified, the extended attributes won't be modified unless a disk is replaced.
+
+It is possible to dump the physical/virtual disks map on the filesystem under the JSON notation to ease or automate disks management. The file path has to be provided using the `--dump-disks-map` command line parameter.
+
+
 ## Anycast IP
 
 The default behavior of the agent is to assign an interface to an IP.
@@ -202,6 +235,7 @@ Tested on:
 * HP ProLiant BL460c Gen8
 * HP ProLiant BL460c Gen9
 * HP ProLiant BL460c Gen10
+* HP ProLiant BL460c Gen10 Graphics Exp its expansion HP ProLiant BL460c Graphics Expansion Blade
 * HP Moonshot 1500 Enclosure (your `DeviceType` should have slots batch create with `Bay c[1-45n1]`) with HP ProLiant m750, m710x, m510 Server Cartridge
 
 ### Pizzas
@@ -254,5 +288,5 @@ On a personal note, I use the docker image from [netbox-community/netbox-docker]
 # git clone https://github.com/netbox-community/netbox-docker
 # cd netbox-docker
 # docker-compose pull
-# docker-compose up 
+# docker-compose up
 ```

default.nix

@@ -0,0 +1,55 @@
+{
+  sources ? import ./npins,
+  pkgs ? import sources.nixpkgs { },
+}:
+
+let
+  checks = (import sources.git-hooks).run {
+    src = ./.;
+
+    hooks =
+      {
+        commitizen.enable = true;
+      }
+      // (pkgs.lib.genAttrs
+        [
+          "black"
+          "isort"
+          "ruff"
+        ]
+        (hook: {
+          enable = true;
+          stages = [ "pre-push" ];
+        })
+      );
+  };
+
+  python3 = pkgs.python3.override {
+    packageOverrides = self: _: {
+      netifaces2 = self.callPackage ./nix/netifaces2.nix { };
+    };
+  };
+in
+
+{
+  devShell = pkgs.mkShell {
+    name = "netbox-agent.dev";
+
+    packages = [
+      (python3.withPackages (ps: [
+        ps.pynetbox
+        ps.netaddr
+        ps.netifaces2
+        ps.pyyaml
+        ps.jsonargparse
+        ps.python-slugify
+        ps.packaging
+        ps.distro
+      ]))
+    ] ++ checks.enabledPackages;
+
+    shellHook = ''
+      ${checks.shellHook}
+    '';
+  };
+}

netbox_agent.yaml.example

@@ -18,14 +18,15 @@ network:
 # blade_role: "Blade"
 # server_role: "Server"
 # tags: server, blade, ,just a comma,delimited,list
-
+# custom_fields: field1=value1,field2=value2
+#
 #
 # Use this to set the tenant
 #
 #tenant:
 # driver: "file:/tmp/tenant"
 # regex: "(.*)"
-
+
 datacenter_location:
  driver: "cmd:cat /etc/qualification | tr [A-Z] [a-z]"
  regex: "datacenter: (?P<datacenter>[A-Za-z0-9]+)"

netbox_agent/__init__.py

@@ -1,6 +1,7 @@
-from pkg_resources import DistributionNotFound, get_distribution
+from importlib.metadata import PackageNotFoundError
+from importlib.metadata import version as _get_version
 
 try:
-    __version__ = get_distribution(__name__).version
-except DistributionNotFound:
+    __version__ = _get_version(__name__)
+except PackageNotFoundError:
     pass

netbox_agent/cli.py

@@ -1,5 +1,8 @@
+from packaging import version
+
 import netbox_agent.dmidecode as dmidecode
 from netbox_agent.config import config
+from netbox_agent.config import netbox_instance as nb
 from netbox_agent.logging import logging  # NOQA
 from netbox_agent.vendors.dell import DellHost
 from netbox_agent.vendors.generic import GenericHost
@@ -9,12 +12,12 @@ from netbox_agent.vendors.supermicro import SupermicroHost
 from netbox_agent.virtualmachine import VirtualMachine, is_vm
 
 MANUFACTURERS = {
-    'Dell Inc.': DellHost,
-    'HP': HPHost,
-    'HPE': HPHost,
-    'Supermicro': SupermicroHost,
-    'Quanta Cloud Technology Inc.': QCTHost,
-    'Generic': GenericHost,
+    "Dell Inc.": DellHost,
+    "HP": HPHost,
+    "HPE": HPHost,
+    "Supermicro": SupermicroHost,
+    "Quanta Cloud Technology Inc.": QCTHost,
+    "Generic": GenericHost,
 }
 
 
@@ -23,26 +26,38 @@ def run(config):
 
     if config.virtual.enabled or is_vm(dmi):
         if not config.virtual.cluster_name:
-            raise Exception('virtual.cluster_name parameter is mandatory because it\'s a VM')
+            raise Exception(
+                "virtual.cluster_name parameter is mandatory because it's a VM"
+            )
         server = VirtualMachine(dmi=dmi)
     else:
-        manufacturer = dmidecode.get_by_type(dmi, 'Chassis')[0].get('Manufacturer')
+        manufacturer = dmidecode.get_by_type(dmi, "Chassis")[0].get("Manufacturer")
         try:
             server = MANUFACTURERS[manufacturer](dmi=dmi)
         except KeyError:
             server = GenericHost(dmi=dmi)
 
+    if version.parse(nb.version) < version.parse("3.7"):
+        print("netbox-agent is not compatible with Netbox prior to version 3.7")
+        return False
+
+    if (
+        config.register
+        or config.update_all
+        or config.update_network
+        or config.update_location
+        or config.update_inventory
+        or config.update_psu
+    ):
+        server.netbox_create_or_update(config)
     if config.debug:
         server.print_debug()
-    if config.register or config.update_all or config.update_network or config.update_location or \
-       config.update_inventory or config.update_psu:
-        server.netbox_create_or_update(config)
     return True
 
 
 def main():
-    return run(config)
+    return 0 if run(config) else 1
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     main()

netbox_agent/config.py

@@ -10,83 +10,171 @@ import urllib3
 def get_config():
     p = jsonargparse.ArgumentParser(
         default_config_files=[
-            '/etc/netbox_agent.yaml',
-            '~/.config/netbox_agent.yaml',
-            '~/.netbox_agent.yaml',
+            "/etc/netbox_agent.yaml",
+            "~/.config/netbox_agent.yaml",
+            "~/.netbox_agent.yaml",
         ],
-        prog='netbox_agent',
+        prog="netbox_agent",
         description="Netbox agent to run on your infrastructure's servers",
-        env_prefix='NETBOX_AGENT_',
-        default_env=True
+        env_prefix="NETBOX_AGENT_",
+        default_env=True,
     )
-    p.add_argument('-c', '--config', action=jsonargparse.ActionConfigFile)
+    p.add_argument("-c", "--config", action=jsonargparse.ActionConfigFile)
 
-    p.add_argument('-r', '--register', action='store_true', help='Register server to Netbox')
-    p.add_argument('-u', '--update-all', action='store_true', help='Update all infos in Netbox')
-    p.add_argument('-d', '--debug', action='store_true', help='Print debug infos')
-    p.add_argument('--update-network', action='store_true', help='Update network')
-    p.add_argument('--update-inventory', action='store_true', help='Update inventory')
-    p.add_argument('--update-location', action='store_true', help='Update location')
-    p.add_argument('--update-psu', action='store_true', help='Update PSU')
+    p.add_argument(
+        "-r", "--register", action="store_true", help="Register server to Netbox"
+    )
+    p.add_argument(
+        "-u", "--update-all", action="store_true", help="Update all infos in Netbox"
+    )
+    p.add_argument("-d", "--debug", action="store_true", help="Print debug infos")
+    p.add_argument("--update-network", action="store_true", help="Update network")
+    p.add_argument("--update-inventory", action="store_true", help="Update inventory")
+    p.add_argument("--update-location", action="store_true", help="Update location")
+    p.add_argument("--update-psu", action="store_true", help="Update PSU")
+    p.add_argument(
+        "--purge-old-devices",
+        action="store_true",
+        help="Purge existing (old ?) devices having same name but different serial",
+    )
+    p.add_argument(
+        "--expansion-as-device",
+        action="store_true",
+        help="Manage blade expansions as external devices",
+    )
 
-    p.add_argument('--log_level', default='debug')
-    p.add_argument('--netbox.url', help='Netbox URL')
-    p.add_argument('--netbox.token', help='Netbox API Token')
-    p.add_argument('--netbox.ssl_verify', default=True, action='store_true',
-                   help='Disable SSL verification')
-    p.add_argument('--virtual.enabled', action='store_true', help='Is a virtual machine or not')
-    p.add_argument('--virtual.cluster_name', help='Cluster name of VM')
-    p.add_argument('--hostname_cmd', default=None,
-                   help="Command to output hostname, used as Device's name in netbox")
-    p.add_argument('--device.tags', default=r'',
-                   help='tags to use for a host')
-    p.add_argument('--device.blade_role', default=r'Blade',
-                   help='role to use for a blade server')
-    p.add_argument('--device.chassis_role', default=r'Server Chassis',
-                   help='role to use for a chassis')
-    p.add_argument('--device.server_role', default=r'Server',
-                   help='role to use for a server')
-    p.add_argument('--tenant.driver',
-                   help='tenant driver, ie cmd, file')
-    p.add_argument('--tenant.driver_file',
-                   help='tenant driver custom driver file path')
-    p.add_argument('--tenant.regex',
-                   help='tenant regex to extract Netbox tenant slug')
-    p.add_argument('--datacenter_location.driver',
-                   help='Datacenter location driver, ie: cmd, file')
-    p.add_argument('--datacenter_location.driver_file',
-                   help='Datacenter location custom driver file path')
-    p.add_argument('--datacenter_location.regex',
-                   help='Datacenter location regex to extract Netbox DC slug')
-    p.add_argument('--rack_location.driver', help='Rack location driver, ie: cmd, file')
-    p.add_argument('--rack_location.driver_file', help='Rack location custom driver file path')
-    p.add_argument('--rack_location.regex', help='Rack location regex to extract Rack name')
-    p.add_argument('--slot_location.driver', help='Slot location driver, ie: cmd, file')
-    p.add_argument('--slot_location.driver_file', help='Slot location custom driver file path')
-    p.add_argument('--slot_location.regex', help='Slot location regex to extract slot name')
-    p.add_argument('--network.ignore_interfaces', default=r'(dummy.*|docker.*)',
-                   help='Regex to ignore interfaces')
-    p.add_argument('--network.ignore_ips', default=r'^(127\.0\.0\..*|fe80.*|::1.*)',
-                   help='Regex to ignore IPs')
-    p.add_argument('--network.lldp', help='Enable auto-cabling feature through LLDP infos')
-    p.add_argument('--inventory', action='store_true',
-                   help='Enable HW inventory (CPU, Memory, RAID Cards, Disks) feature')
+    p.add_argument("--log_level", default="debug")
+    p.add_argument("--netbox.ssl_ca_certs_file", help="SSL CA certificates file")
+    p.add_argument("--netbox.url", help="Netbox URL")
+    p.add_argument("--netbox.token", help="Netbox API Token")
+    p.add_argument(
+        "--netbox.ssl_verify",
+        default=True,
+        action="store_true",
+        help="Disable SSL verification",
+    )
+    p.add_argument(
+        "--virtual.enabled", action="store_true", help="Is a virtual machine or not"
+    )
+    p.add_argument("--virtual.cluster_name", help="Cluster name of VM")
+    p.add_argument(
+        "--hostname_cmd",
+        default=None,
+        help="Command to output hostname, used as Device's name in netbox",
+    )
+    p.add_argument(
+        "--device.platform",
+        default=None,
+        help="Override device platform. Here we use OS distribution.",
+    )
+    p.add_argument("--device.tags", default=r"", help="tags to use for a host")
+    p.add_argument(
+        "--preserve-tags",
+        action="store_true",
+        help="Append new unique tags, preserve those already present",
+    )
+    p.add_argument(
+        "--device.custom_fields",
+        default=r"",
+        help="custom_fields to use for a host, eg: field1=v1,field2=v2",
+    )
+    p.add_argument(
+        "--device.blade_role", default=r"Blade", help="role to use for a blade server"
+    )
+    p.add_argument(
+        "--device.chassis_role",
+        default=r"Server Chassis",
+        help="role to use for a chassis",
+    )
+    p.add_argument(
+        "--device.server_role", default=r"Server", help="role to use for a server"
+    )
+    p.add_argument("--tenant.driver", help="tenant driver, ie cmd, file")
+    p.add_argument("--tenant.driver_file", help="tenant driver custom driver file path")
+    p.add_argument("--tenant.regex", help="tenant regex to extract Netbox tenant slug")
+    p.add_argument(
+        "--datacenter_location.driver", help="Datacenter location driver, ie: cmd, file"
+    )
+    p.add_argument(
+        "--datacenter_location.driver_file",
+        help="Datacenter location custom driver file path",
+    )
+    p.add_argument(
+        "--datacenter_location.regex",
+        help="Datacenter location regex to extract Netbox DC slug",
+    )
+    p.add_argument("--rack_location.driver", help="Rack location driver, ie: cmd, file")
+    p.add_argument(
+        "--rack_location.driver_file", help="Rack location custom driver file path"
+    )
+    p.add_argument(
+        "--rack_location.regex", help="Rack location regex to extract Rack name"
+    )
+    p.add_argument("--slot_location.driver", help="Slot location driver, ie: cmd, file")
+    p.add_argument(
+        "--slot_location.driver_file", help="Slot location custom driver file path"
+    )
+    p.add_argument(
+        "--slot_location.regex", help="Slot location regex to extract slot name"
+    )
+    p.add_argument(
+        "--network.ignore_interfaces",
+        default=r"(dummy.*|docker.*)",
+        help="Regex to ignore interfaces",
+    )
+    p.add_argument(
+        "--network.ignore_ips",
+        default=r"^(127\.0\.0\..*|fe80.*|::1.*)",
+        help="Regex to ignore IPs",
+    )
+    p.add_argument(
+        "--network.ipmi", default=True, help="Enable gathering IPMI information"
+    )
+    p.add_argument(
+        "--network.lldp", help="Enable auto-cabling feature through LLDP infos"
+    )
+    p.add_argument(
+        "--inventory",
+        action="store_true",
+        help="Enable HW inventory (CPU, Memory, RAID Cards, Disks) feature",
+    )
+    p.add_argument(
+        "--process-virtual-drives",
+        action="store_true",
+        help="Process virtual drives information from RAID "
+        "controllers to fill disk custom_fields",
+    )
+    p.add_argument(
+        "--force-disk-refresh",
+        action="store_true",
+        help="Forces disks detection reprocessing",
+    )
+    p.add_argument(
+        "--dump-disks-map", help="File path to dump physical/virtual disks map"
+    )
 
     options = p.parse_args()
     return options
 
 
+config = get_config()
+
+
 def get_netbox_instance():
-    config = get_config()
     if config.netbox.url is None or config.netbox.token is None:
-        logging.error('Netbox URL and token are mandatory')
+        logging.error("Netbox URL and token are mandatory")
         sys.exit(1)
 
     nb = pynetbox.api(
         url=get_config().netbox.url,
         token=get_config().netbox.token,
     )
-    if get_config().netbox.ssl_verify is False:
+    ca_certs_file = config.netbox.ssl_ca_certs_file
+    if ca_certs_file is not None:
+        session = requests.Session()
+        session.verify = ca_certs_file
+        nb.http_session = session
+    elif config.netbox.ssl_verify is False:
         urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
         session = requests.Session()
         session.verify = False
@@ -95,5 +183,4 @@ def get_netbox_instance():
     return nb
 
 
-config = get_config()
 netbox_instance = get_netbox_instance()

netbox_agent/dmidecode.py

@@ -5,55 +5,57 @@ import sys
 
 from netbox_agent.misc import is_tool
 
-_handle_re = _re.compile('^Handle\\s+(.+),\\s+DMI\\s+type\\s+(\\d+),\\s+(\\d+)\\s+bytes$')
-_in_block_re = _re.compile('^\\t\\t(.+)$')
-_record_re = _re.compile('\\t(.+):\\s+(.+)$')
-_record2_re = _re.compile('\\t(.+):$')
+_handle_re = _re.compile(
+    "^Handle\\s+(.+),\\s+DMI\\s+type\\s+(\\d+),\\s+(\\d+)\\s+bytes$"
+)
+_in_block_re = _re.compile("^\\t\\t(.+)$")
+_record_re = _re.compile("\\t(.+):\\s+(.+)$")
+_record2_re = _re.compile("\\t(.+):$")
 
 _type2str = {
-    0: 'BIOS',
-    1: 'System',
-    2: 'Baseboard',
-    3: 'Chassis',
-    4: 'Processor',
-    5: 'Memory Controller',
-    6: 'Memory Module',
-    7: 'Cache',
-    8: 'Port Connector',
-    9: 'System Slots',
-    10: ' On Board Devices',
-    11: ' OEM Strings',
-    12: ' System Configuration Options',
-    13: ' BIOS Language',
-    14: ' Group Associations',
-    15: ' System Event Log',
-    16: ' Physical Memory Array',
-    17: ' Memory Device',
-    18: ' 32-bit Memory Error',
-    19: ' Memory Array Mapped Address',
-    20: ' Memory Device Mapped Address',
-    21: ' Built-in Pointing Device',
-    22: ' Portable Battery',
-    23: ' System Reset',
-    24: ' Hardware Security',
-    25: ' System Power Controls',
-    26: ' Voltage Probe',
-    27: ' Cooling Device',
-    28: ' Temperature Probe',
-    29: ' Electrical Current Probe',
-    30: ' Out-of-band Remote Access',
-    31: ' Boot Integrity Services'