Merge pull request 'Fix disk size inventory' (#310) from ribetm/disk_size into main

Reviewed-on: #310

.envrc

@@ -0,0 +1 @@
+use nix

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,25 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Configuration file**
+Paste here your netbox agent configuration file
+
+**Environment:**
+ - OS: [e.g. Ubuntu 18.04]
+ - Netbox agent version [e.g. master, v0.6.0]
+
+**Additional context**
+Add any other context about the problem here.

.gitignore

@@ -179,3 +179,8 @@ dmypy.json
 .pyre/
 
 # End of https://www.gitignore.io/api/emacs,python
+
+netbox-docker
+/.vscode
+.direnv
+.pre-commit-config.yaml

.travis.yml

@@ -2,20 +2,23 @@ sudo: false
 dist: xenial
 language: python
 
+services:
+  - docker
+
 matrix:
     include:
         - python: 3.5
-          env: TOXENV=py35
+          env: TOXENV=pytest
         - python: 3.6
-          env: TOXENV=py36
+          env: TOXENV=pytest
         - python: 3.7
-          env: TOXENV=py37
+          env: TOXENV=pytest
         - python: pypy3
-          env: TOXENV=pypy3
+          env: TOXENV=pytest
         - python: 3.5
-          env: TOXENV=pep8
+          env: TOXENV=flake8
         - python: 3.7
-          env: TOXENV=pep8
+          env: TOXENV=flake8
 
 cache:
   directories:
@@ -23,7 +26,7 @@ cache:
 
 install:
   - pip install tox
-  - if [[ $TOXENV == py* ]]; then pip install coveralls; fi
+  - if [[ $TOXENV == pytest ]]; then pip install coveralls; fi
 
 script:
   - tox
@@ -32,4 +35,4 @@ notifications:
   email: false
 
 after_success:
-  - if [[ $TOXENV == py* ]]; then coveralls; fi
+  - if [[ $TOXENV == pytest ]]; then coveralls; fi

MANIFEST.in

@@ -0,0 +1 @@
+include requirements.txt

README.md

@@ -1,5 +1,4 @@
-# Netbox agent
-
+# Netbox agent [![Build Status](https://travis-ci.com/Solvik/netbox-agent.svg?branch=master)](https://travis-ci.com/Solvik/netbox-agent)
 
 This project aims to create hardware automatically into [Netbox](https://github.com/netbox-community/netbox) based on standard tools (dmidecode, lldpd, parsing /sys/, etc).
 
@@ -7,21 +6,22 @@ The goal is to generate an existing infrastructure on Netbox and have the abilit
 
 # Features
 
-* Create servers, chassis and blade through standard tools (`dmidecode`)
+* Create virtual machines, servers, chassis and blade through standard tools (`dmidecode`)
 * Create physical, bonding and vlan network interfaces with IPs (IPv4 & IPv6)
 * Create IPMI interface if found
 * Create or get existing VLAN and associate it to interfaces
 * Generic ability to guess datacenters and rack location through drivers (`cmd` and `file` and custom ones)
 * Update existing `Device` and `Interface`
 * Handle blade moving (new slot, new chassis)
+* Handle blade GPU expansions
 * Automatic cabling (server's interface to switch's interface) using lldp
-* Local inventory using `Inventory Item` for CPU, RAM, RAID cards, physical disks (behind raid cards)
+* Local inventory using `Inventory Item` for CPU, GPU, RAM, RAID cards, physical disks (behind raid cards)
 * PSUs creation and power consumption reporting (based on vendor's tools)
 
 # Requirements
 
-- Netbox >= 2.6
-- Python >= 3.4
+- Netbox >= 3.7
+- Python >= 3.7
 - [pynetbox](https://github.com/digitalocean/pynetbox/)
 - [python3-netaddr](https://github.com/drkjam/netaddr)
 - [python3-netifaces](https://github.com/al45tair/netifaces)
@@ -33,35 +33,112 @@ The goal is to generate an existing infrastructure on Netbox and have the abilit
 - lldpd
 - lshw
 
-# Known limitations
+## Inventory requirement
+- hpassacli
+- storcli
+- omreport
 
-* The project is only compatible with Linux.
-Since it uses `ethtool` and parses `/sys/` directory, it's not compatible with *BSD distributions.
-* Netbox `>=2.6.0,<=2.6.2` has a caching problem ; if the cache lifetime is too high, the script can get stale data after modification.
-We advise to set `CACHE_TIME` to `0`.
+# Installation
+
+```
+# pip3 install netbox-agent
+```
+
+# Usage
+
+The agent can be run from a shell and get its configuration from either the configuration file or environment variables.
+
+Configuration values are overridden based on the following precedence: command line arguments (might include config file) > environment variables > default config file > defaults.
+
+```
+# netbox_agent -c /etc/netbox_agent.yaml --register
+INFO:root:Creating chassis blade (serial: QTFCQ574502EF)
+INFO:root:Creating blade (serial: QTFCQ574502D2) myserver on chassis QTFCQ574502EF
+INFO:root:Setting device (QTFCQ574502D2) new slot on Slot 9 (Chassis QTFCQ574502EF)..
+INFO:root:Interface a8:1e:84:f2:9e:6a not found, creating..
+INFO:root:Creating NIC enp1s0f1 (a8:1e:84:f2:9e:6a) on myserver
+INFO:root:Interface 02:42:7a:89:cf:a4 not found, creating..
+INFO:root:Creating NIC br-07ea1e4a2f0e (02:42:7a:89:cf:a4) on myserver
+INFO:root:Create new IP 172.19.0.1/16 on br-07ea1e4a2f0e
+INFO:root:Interface a8:1e:84:f2:9e:69 not found, creating..
+INFO:root:Creating NIC enp1s0f0 (a8:1e:84:f2:9e:69) on myserver
+INFO:root:Create new IP 42.42.42.42/24 on enp1s0f0
+INFO:root:Create new IP fe80::aa1e:84ff:fef2:9e69/64 on enp1s0f0
+INFO:root:Interface a8:1e:84:cd:9d:d6 not found, creating..
+INFO:root:Creating NIC IPMI (a8:1e:84:cd:9d:d6) on myserver
+INFO:root:Create new IP 10.191.122.10/24 on IPMI
+```
+
+If you need, you can update only specific informations like:
+* Network
+* Inventory
+* Location
+* PSUs
+
+```
+# ip a add 42.42.42.43/24 dev enp1s0f1
+# netbox_agent -c /etc/netbox_agent.yaml --update-network
+INFO:root:Create new IP 42.42.42.43/24 on enp1s0f1
+# netbox_agent --update-inventory
+INFO:root:Creating Disk Samsung SSD 850 S2RBNX0K101698D
+```
 
 # Configuration
 
 ```
+# Netbox configuration
 netbox:
  url: 'http://netbox.internal.company.com'
  token: supersecrettoken
+ # uncomment to disable ssl verification
+ # ssl_verify: false
+ # uncomment to use the system's CA certificates
+ # ssl_ca_certs_file: /etc/ssl/certs/ca-certificates.crt
 
+# Network configuration
 network:
+  # Regex to ignore interfaces
   ignore_interfaces: "(dummy.*|docker.*)"
+  # Regex to ignore IP addresses
   ignore_ips: (127\.0\.0\..*)
-  # enable auto-cabling
+  # enable auto-cabling by parsing LLDP answers
   lldp: true
 
+#
+# You can use these to change the Netbox roles.
+# These are the defaults.
+#
+#device:
+# chassis_role: "Server Chassis"
+# blade_role: "Blade"
+# server_role: "Server"
+# tags: server, blade, ,just a comma,delimited,list
+# custom_fields: field1=value1,field2=value2#
+#
+# Can use this to set the tenant
+#
+#tenant:
+# driver: "file:/tmp/tenant"
+# regex: "(.*)"
+
+## Enable virtual machine support
+# virtual:
+#   # not mandatory, can be guessed
+#   enabled: True
+#   # see https://netbox.company.com/virtualization/clusters/
+#   cluster_name: my_vm_cluster
+
+# Enable datacenter location feature in Netbox
 datacenter_location:
- driver: "cmd:cat /etc/qualification | tr [a-z] [A-Z]"
- regex: "DATACENTER: (?P<datacenter>[A-Za-z0-9]+)"
+ driver: "cmd:cat /etc/qualification | tr [A-Z] [a-z]"
+ regex: "datacenter: (?P<datacenter>[A-Za-z0-9]+)"
 # driver: 'cmd:lldpctl'
 # regex: 'SysName: .*\.([A-Za-z0-9]+)'
 #
 # driver: "file:/tmp/datacenter"
 # regex: "(.*)"
 
+# Enable rack location feature in Netbox
 rack_location:
 # driver: 'cmd:lldpctl'
 # match SysName: sw-dist-a1.dc42
@@ -70,6 +147,7 @@ rack_location:
 # driver: "file:/tmp/datacenter"
 # regex: "(.*)"
 
+# Enable local inventory reporting
 inventory: true
 ```
 
@@ -84,6 +162,36 @@ The `get_blade_slot` method return the name of the `Device Bay`.
 
 Certain vendors don't report the blade slot in `dmidecode`, so we can use the `slot_location` regex feature of the configuration file.
 
+Some blade servers can be equipped with additional hardware using expansion blades, next to the processing blade, such as GPU expansion, or drives bay expansion. By default, the hardware from the expnasion is associated with the blade server itself, but it's possible to register the expansion as its own device using the `--expansion-as-device` command line parameter, or by setting `expansion_as_device` to `true` in the configuration file.
+
+## Drives attributes processing
+
+It is possible to process drives extended attributes such as the drive's physical or logical identifier, logical drive RAID type, size, consistency and so on.
+
+Those attributes as set as `custom_fields` in Netbox, and need to be registered properly before being able to specify them during the inventory phase.
+
+As the custom fields have to be created prior being able to register the disks extended attributes, this feature is only activated using the `--process-virtual-drives` command line parameter, or by setting `process_virtual_drives` to `true` in the configuration file.
+
+The custom fields to create as `DCIM > inventory item` `Text` are described below.
+
+```
+NAME            LABEL                      DESCRIPTION
+mount_point     Mount point                Device mount point(s)
+pd_identifier   Physical disk identifier   Physical disk identifier in the RAID controller
+vd_array        Virtual drive array        Virtual drive array the disk is member of
+vd_consistency  Virtual drive consistency  Virtual disk array consistency
+vd_device       Virtual driv