Remove the -l command like option from run-tests.py and always enable
writing of debug level logs to files. The stdout debug verbosity is
controlled independently of the debug log files.
Signed-hostap: Jouni Malinen <j@w1.fi>
This is needed to avoid invalid PBC session overlap detection when
the previous test case used active PBC mode and the old BSS entry
in cfg80211 may still be valid when starting the next test case.
Signed-hostap: Jouni Malinen <j@w1.fi>
A full scan in these specific test cases does not add anything to the
coverage, so use a single channel scan for the station connection to
remove undesired extra time needed for a full scan.
Signed-hostap: Jouni Malinen <j@w1.fi>
This optional argument can be used to randomize the order in which the
test cases are run. This can provide more coverage on testing
interactions of common use cases in various different sequences. Such
issues have already been found even with the fixed order of test cases,
but being able to reorder the tests makes this more efficient.
Signed-hostap: Jouni Malinen <j@w1.fi>
This is easier than to copy/paste from the README. We may
want to extend it later to change the .config for some
common differences between systems (e.g., libnl/libbfd).
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
In some cases, e.g., with the VM tests if the VM crashes, it
can be useful to know which tests should have run but didn't
(or didn't finish). In order to catch these more easily, add
an option to prefill the database with all tests at the very
beginning of the testing (in a new NOTRUN state) and use the
option in the VM tests.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
Create a results.db in the output directory when running
the tests in a VM. To make that easier, create the tables
in the python script if they don't exist.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
Refactor the test reporting to treat the different results
(success/skip/failure) identically. This makes the timing
seem a bit longer, but cleans up the code which will allow
for adding more checks (e.g., on the captured data files)
later.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
Rather than just having KERNELDIR, allow setting KERNEL directly.
Also remove the -s option that prevents running multiple machines
at the same time, but add a KVMARGS= variable that can be used to
restore that if needed.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
test_ap_bss_add_remove verifies hostapd behavior when BSSes are
added/removed in multi-BSS configuration.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
This allows run-tests.py to use the same logs/<date> default logdir as
start.sh which is quite convenient for manual test runs.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
The run-tests.py -l argument does not take an argument value anymore.
Instead, debug output is directed to a separate file <test>.log for each
test case.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
This is unnecessary extra complexity for user, so use the 'test_' prefix
only internally within the python scripts and file names.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
This is unnecessary extra complexity for user and reports, so use the
'test_' prefix only internally within the python scripts.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
There is no need to run a separate tcpdump process to capture frames
from hwsim0 since wlantest is already doing that can write the results
to a file.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
This file was used for adding debug info into the buildbot logs. The new
sqlite database-based design will replace that, so there is no need to
create last-debug and getting rid of it will make it easier to split the
run log into per-test case files.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
Commit 781b65cfbb ended up accidentally
changing this from an integer to a string. Fix this by not converting
the variable into a string.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
Instead of running on the host, it can be useful to run in a
VM, particularly to test kernel rather than userspace changes,
so add a few scripts that allow doing so easily.
The basic idea is that the VM kernel is the same architecture
as the host kernel, so the host's root filesystem can be used
(in read-only mode) to run everything. Only a log filesystem
is mounted read-write and will get all the test output.
The kernel console output is collected to a special 'console'
file in the logs directory and kernel crashes are detected.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
If running in a VM with the logs going to a host filesystem
chown isn't actually possible, silence any warnings from it.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
Resetting at the beginning causes the reset logging/tracing
data to leak from the previous test into the next, and the
last one being missed at all - reset at the end of each run
instead. Also reset before all tests just in case running a
test actually crashed the python script.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
In addition to tracing, allow collecting dmesg. There's no
provision for actually looking at it and finding problems
in it yet though.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
Instead of passing the log directory for each option
(-l, -r, -e, and -T) pass it once and make the other
options just take the filename (optionally, even).
This will also make it easier to extend later.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
Instead of logging many files called "<timestamp>-*", log into
a new "<timestamp>/*" in the logs/ subdirectory and only put
the last-debug file into logs/. If a LOGDIR is specified in the
environment, instead just put everything into that directory
(so the caller should make sure to give it a timestamp or so)
and skip the creation of last-debug entirely.
Also clean up a bit and pass the LOGDIR from run-all.sh to
start.sh rather than having start.sh create the timestamp and
run-all.sh detect the latest one when having run start.sh.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
In order to get tracing per test, allow run-tests.py to start
and stop tracing per test case. This is implemented using a
python 'with' context so it starts/stops automatically at the
right spots.
Instead of starting global tracing, also use it from run-all.sh
and put the trace files into the log dir.
Note that this only works right if you use a separate log dir
for all test runs as the trace files aren't timestamped.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
Just like wpa_supplicant, give hostapd the -T option to
send all debug messages into the Linux tracing buffer.
Enable this option for hwsim test builds by default.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
Allow run-all to pass on extra arguments from the command
line that aren't built by the script itself.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
Reuse the code rather than duplicating the implementation
of starting the tests. To make that easier, allow passing
multiple modules with -f to run-tests.py.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
Instead of hard-coding four different cases, use variables
(and printf) to reduce the duplication.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
Instead of re-implementing a command-line parser, use the
argparse module.
The only real change (I hope) is that the test module must
now be given to the -f option without the .py suffix.
Also, --help now works, and if a test module/test name is
given that doesn't exist, the valid list is printed.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
Instead of hardcoding logs/, allow putting LOGDIR into
the environment. This helps run tests in a VM where the
rootfs might be mounted read-only.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
test_ap_change_ssid verifies that the SET and RELOAD control interface
commands can be used to change the SSID of a hostapd-controlled WPA2-PSK
AP, i.e., for a case where the PSK needs to be derived again after the
SSID change.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
The test_ap_hs20_username* test cases verify that a username/password
credential can be used for Hotspot 2.0 connection and that the network
type is reported correctly.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
test_ap_hs20_ext_sim_roaming is otherwise identical with
test_ap_hs20_ext_sim, but the network is operated by a roaming partner.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
This verifies that the network type (home/roaming/unknown) is available
and set properly in the test_ap_hs20_ext_sim test case.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
Since the scripts must be run from the source directory to
find the tests to run, they can use a relative path to the
wpaspy module instead of requiring it to be installed.
Signed-hostap: Johannes Berg <johannes.berg@intel.com>
"run-tests.py -S <db file> -L" can now be used to update a database
table with the current set of test cases and their descriptions.
Signed-hostap: Jouni Malinen <j@w1.fi>
cfg80211 may keep some state (e.g., regulatory domain), so make sure
this gets cleared between each full testing cycle.
Signed-hostap: Jouni Malinen <j@w1.fi>
test_p2p_go_invite_auth is similar to test_p2p_go_invite with the main
difference being in the peer device pre-authorizing the invitation
instead of processing invitation at upper layers after having received
it.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
There is no point trying to go through a test case if the NOTE command
to write TEST-START entry does not succeed. This avoids some excessive
waits on buildbot trying to forcefully kill the programs on its timeout
if wpa_supplicant gets stuck waiting for something (like the current
issue with libnl events and commands having a chance of hitting a
blocking wait on netlink messages).
Signed-hostap: Jouni Malinen <j@w1.fi>
This adds more coverage to testing by forcing the GO to be found with an
older entry in the BSS table and with that entry having a different
operating channel. Such a case has found issues with PD retries and
scanning if incorrect frequency and SSID is selected for the group.
Instead of relying on the old BSS entry with different operating channel
to happen based on a specific test case sequence, force this to happen
for this test case every time.
Signed-hostap: Jouni Malinen <j@w1.fi>
This verifies network selection using 3GPP Network Information and
connection using EAP-SIM with GSM authentication handled by an external
program.
Signed-hostap: Jouni Malinen <j@w1.fi>
This is in preparation for supporting multiple domain values after which
setting the domain value does not replace the old value, but adds a new
one.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
If a test run is terminated forcefully, hlr_auc_gw could have been left
running and/or some of the control interface files could still remain.
This could result in hlr_auc_gw not starting again for the next test
run.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
Do not fail the EAP-SIM/AKA/AKA' test cases if hlr_auc_gw is not
available since hlr_auc_gw is not considered required part for the test
setup.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
Some older systems used admin group, but adm group seems to be more
common nowadays, so detect this automatically rather than assume admin
group is used.
Signed-hostap: Jouni Malinen <j@w1.fi>
This can be used by test cases that depend on some external component
that may not always be available to indicate clearly that a test case
was skipped rather than passed or failed.
Signed-hostap: Jouni Malinen <j@w1.fi>
There is no point trying to go through the tests if hostapd and
wpa_supplicant processes cannot be started properly.
Signed-hostap: Jouni Malinen <j@w1.fi>
cfg80211/mac80211 seems to getting stuck with scans every now and then.
Check for this special state and delay return from reset() until the
driver has stopped the scan operation. This reduces likelihood of
failing multiple test cases in a row because of a single error.
Signed-hostap: Jouni Malinen <j@w1.fi>
test_p2p_go_invite verifies that GO can invite a peer in a case where
the operating channel and peer's listen channel are different.
Signed-hostap: Jouni Malinen <j@w1.fi>
run-all.sh and start.sh scripts can now take 'trace' command line
argument to request Linux tracing information from mac80211, cfg80211,
and wpa_supplicant to be recorded.
Signed-hostap: Jouni Malinen <j@w1.fi>
This avoids some issues with the global control interface socket
getting stuck with send buffer limit reached if test cases did not
use wait_global_event().
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
Some test runs have resulted in wpa_supplicant being unresponsive or
very slow on the control interface. Check for that type of issues as
part of the reset sequence that is done between each test case.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
test_persistent_group_invite_removed_client verifies that a client
that is removed from a group can rejoin based on new invitation.
Signed-hostap: Jouni Malinen <j@w1.fi>
Use test_autogo to verify that P2P client is able to notice explicit
group session termination message from the GO. This would have caught
the regression in processing P2P deauthentication notifications.
Signed-hostap: Jouni Malinen <j@w1.fi>
This is another one of the test cases that can time out frequently
under valgrind during WPS exchange. Increase the timeout to make
false error reports less likely to occur.
Signed-hostap: Jouni Malinen <j@w1.fi>
Number of crypto operations seem to take very long time in the valgrind
tests (about five seconds for public key generation for M1 and M2 on a
virtual server) and this is enough to push the test runs to hit the
timeout frequently even when there is no real error. Make this less
frequent by increasing WPS test case timeout from 15 to 30 seconds to
avoid issues based on the test scripts.
Signed-hostap: Jouni Malinen <j@w1.fi>
Number of crypto operations seem to take very long time in the valgrind
tests (about five seconds for passphrase to PSK mapping and for public
key generation for M1 and M2 on a virtual server) and this is enough to
push the test runs to hit the timeout frequently even when there is no
real error. Make this less frequent by increasing group formation
timeout from 15 to 20 seconds to avoid issues based on the test scripts
(15 + config time seconds timeout in the protocol may still kick in,
though).
Signed-hostap: Jouni Malinen <j@w1.fi>
Wait a bit longer for wpa_supplicant/hostapd to exit and clear control
interface files separately if either any of the cleanup steps fail to do
so.
Signed-hostap: Jouni Malinen <j@w1.fi>
Instead of waiting for a fixed duration, poll hostapd/wpa_supplicant
through the control interface until they are ready to reply in order to
avoid starting test case execution too early. This is like commit
b76579e0c0 but for the concurrent P2P test
cases that were forgotten from that commit.
Signed-hostap: Jouni Malinen <j@w1.fi>
It looks like mac80211 does not always manage to join the same IBSS and
this can result in test_ibss_rsn failures. Detect this case and try to
get the IBSSes to merge by running a scan prior to continuing with the
test case.
Signed-hostap: Jouni Malinen <j@w1.fi>
The handshake between STA2 and STA1 was not tracked previously. For
completeness, include that in the list of handshakes that are explicitly
waited for to complete prior to starting the connectivity tests. Since
all key setup should be done at this point, remove the extra wait before
the last connectivity test between STA1 and STA2.
Signed-hostap: Jouni Malinen <j@w1.fi>
It looks like there may not be enough time to complete this test case in
15 seconds under valgrind on a virtual host every time, so increase the
timeout to 30 seconds to get more consistent test results.
Signed-hostap: Jouni Malinen <j@w1.fi>
Instead of waiting for a fixed duration, poll hostapd/wpa_supplicant
through the control interface until they are ready to reply in order to
avoid starting test case execution too early.
Signed-hostap: Jouni Malinen <j@w1.fi>
Try to make sure new testing can be started by forcing
hostapd/wpa_supplicant to be killed with SIGKILL if needed. In addition,
wait a bit longer when killing the processes to avoid issues with the
next test run starting before the old one has been fully terminated.
Signed-hostap: Jouni Malinen <j@w1.fi>
The previous design of having to provide valgrind parameter to
stop-wifi.sh did not work since the new test setup may no match the
previous one. Instead of trying to figure out what needs to be stopped
based on previous run, stop all wpa_supplicant/hostapd processes running
under valgrind unconditionally to avoid cases where new set of processes
cannot be started due to existing processed.
Signed-hostap: Jouni Malinen <j@w1.fi>
test_ibss_rsn has been failing pretty frequently due to various issues
in getting all connections up. Try to address this initially by avoiding
concurrent operations and explicitly waiting for 4-way handshake
completion before testing data connection. Once this test case is
workign more robustly, a more difficult case with concurrent operations
can be added as a separate test case.
Signed-hostap: Jouni Malinen <j@w1.fi>
This makes it easier to figure out what failed and allows builbot to
integrate multiple logs and state information about the test cases.
Signed-hostap: Jouni Malinen <j@w1.fi>
It looks like some virtual host setups are too slow to get
wpa_supplicant initialized under valgrind and this was triggering test
failures. Work around this by using even a longer wait before test case
execution.
Signed-hostap: Jouni Malinen <j@w1.fi>
It looks like valgrind could not under buildbot with the default file
mode. Fix this by making wpa_supplicant and hostapd binaries readable
and executable by everyone.
Signed-hostap: Jouni Malinen <j@w1.fi>
./run-all.sh can now take an optional argument to select whether to run
all test cases as before (default), to run these under valgrind
("valgrind"), to run P2P test cases with concurrent station interface
("concurrent"), or the concurrent P2P test cases under valgrind
("concurrent-valgrind").
valgrind cases report errors if a test case fails or valgrind reports an
error.
Signed-hostap: Jouni Malinen <j@w1.fi>
print and logger.info() were directing output to different locations
(stdout and stderr, respectively) which resulted in buildbot showing
reordered entries. Use logger consistently to avoid that.
Signed-hostap: Jouni Malinen <j@w1.fi>
It looks like the NOTE commands can time out in some cases. Avoid
stopping the test run in such a case to get more coverage if this is a
temporary issue.
Signed-hostap: Jouni Malinen <j@w1.fi>
The design for launching AP mode interfaces was changed, but
start-p2p-concurrent.sh was not updated with matching changes.
Signed-hostap: Jouni Malinen <j@w1.fi>
Make sure the connection between each STA and the AP is in working state
between each TDLS setup/teardown operation.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
This allows more consistent interface to be used regardless of which
P2P driver design is used (especially for P2P management operations
over netdev vs. dedicated P2P_DEVICE).
Signed-hostap: Jouni Malinen <j@w1.fi>
The scan results from old test cases showing different SSID on the AP
can result in false failures, so avoid those by ignoring old scan
results from cfg80211.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
Stopping the AP first was not ideal for the test cases since it could
result in wpa_supplicant trying to connect back and start a scan at the
end of a test case and cause problems for the following test case that
tried to scan in the beginning while the previously started scan was
still in progress.
Signed-hostap: Jouni Malinen <j@w1.fi>
Do not print the potentially long list of passed test cases. In case of
failure(s), make sure the failed test list is the last item in the
report.
Signed-hostap: Jouni Malinen <j@w1.fi>
The WPS tests are more prone to fail if scan results from the previous
test cases are allowed to remain in the wpa_supplicant BSS table during
the consecutive test since the test setup uses the same BSSID for the
test APs that change their configuration. Avoid these mostly bogus
issues by enforcing wpa_supplicant to drop and ignore old scan results
during the WPS test cases.
Signed-hostap: Jouni Malinen <j@w1.fi>
This removes the unnecessary separation of P2P (no hostapd) and AP
tests. The same scripts can be used to prepare for these tests and to
execute the tests.
Signed-hostap: Jouni Malinen <j@w1.fi>
P2P_PROV_DISC fails if the peer is not fully discovered and the previous
version of the test cases did not force this (i.e., it could have
started PD just based on having seen a Probe Request frame from the
peer).
Signed-hostap: Jouni Malinen <j@w1.fi>
buildbot seems to be running with more limited umask by default and that
prevented the hostapd per-interface socket from being accessed.
Signed-hostap: Jouni Malinen <j@w1.fi>
A virtual station interface in scanning state can change the P2P
operations quite a bit, so running all the P2P tests also in such
setup can increase testing coverage quite a bit.
Signed-hostap: Jouni Malinen <j@w1.fi>
The discovery and group formation test cases do not need to run the full
scan as the first step, so get rid of it to make the tests complete more
quickly.
Signed-hostap: Jouni Malinen <jouni@qca.qualcomm.com>
This is the most common sequence for initiating GO Negotiation, so
include a test case that uses threads to follow both peers at the
same time to allow GO Negotiation to be re-initiated.
Signed-hostap: Jouni Malinen <j@w1.fi>
Command line parameter to run-p2p-tests.py can now be used to select
which test case is run instead of always running all test cases.
Signed-hostap: Jouni Malinen <j@w1.fi>
run-p2p-tests.py can now be used to run all P2P test cases. The
actual test cases are defined in test_p2p_*.py files.
Signed-hostap: Jouni Malinen <j@w1.fi>
This will hopefully grow over time to become a much more complete
testing mechanism that uses mac80211_hwsim to verify various
wpa_supplicant and hostapd functions automatically.
Signed-hostap: Jouni Malinen <j@w1.fi>
