summaryrefslogtreecommitdiff
path: root/test/py/u_boot_console_base.py
Commit message (Collapse)AuthorAgeLines
* test/py: match prompt only at line boundariesStephen Warren2016-08-20-3/+3
| | | | | | | | | | | This prevents capture of command output from terminating early on boards that use a simple prompt (e.g. "=> ") that appears in the middle of command output (e.g. crc32's "... ==> 2fa737e0"). Reported-by: Tom Rini <trini@konsulko.com> Signed-off-by: Stephen Warren <swarren@wwwdotorg.org> Tested-by: Tom Rini <trini@konsulko.com> Reviewed-by: Heiko Schocher <hs@denx.de>
* test: Adjust run_command_list() to return a list of stringsSimon Glass2016-08-05-3/+4
| | | | | | | | Return one string for each command that was executed. This seems cleaner. Suggested-by: Teddy Reed <teddy.reed@gmail.com> Signed-off-by: Simon Glass <sjg@chromium.org> Reviewed-by: Stephen Warren <swarren@nvidia.com>
* test: Add a function to restart U-BootSimon Glass2016-08-05-0/+5
| | | | | | | | | | | Add a proper function for this rather than using internal functions. Use it in the single call site. Also, do a restart at the end of the vboot test to reset to the normal device tree. Signed-off-by: Simon Glass <sjg@chromium.org> Suggested-by: Stephen Warren <swarren@nvidia.com>
* test: Fix typos in commentsSimon Glass2016-08-05-2/+2
| | | | | | | | Fix some typos in various files introduced with the vboot test conversion. Reported-by: Teddy Reed <teddy.reed@gmail.com> Signed-off-by: Simon Glass <sjg@chromium.org>
* Merge git://git.denx.de/u-boot-dmTom Rini2016-07-15-1/+11
|\
| * test/py: Provide a way to get early console outputSimon Glass2016-07-14-0/+10
| | | | | | | | | | | | | | Some tests want to check the console output from SPL or U-Boot proper. Provide a means to do this. Signed-off-by: Simon Glass <sjg@chromium.org>
| * test/py: Note which console produced unexpected outputSimon Glass2016-07-14-1/+1
| | | | | | | | | | | | | | | | At present the SPL and U-Boot consoles both present the same error message when the expected console output does not appear. Add "SPL" to the SPL error message to resolve this ambiguity. Signed-off-by: Simon Glass <sjg@chromium.org>
* | test/py: Add a helper to run a list of U-Boot commandsSimon Glass2016-07-14-0/+16
|/ | | | | | | Some tests want to execute a sequence of commands. Add a helper for this. Signed-off-by: Simon Glass <sjg@chromium.org> Reviewed-by: Teddy Reed <teddy.reed@gmail.com>
* test/py: Support setting up specific timeoutMichal Simek2016-05-27-0/+31
| | | | | | | | | | | | | | Large file transfers, flash erasing and more complicated tests requires more time to finish. Provide a way to setup specific timeout directly in test. For example description for 50s test: timeout = 50000 with u_boot_console.temporary_timeout(timeout): u_boot_console.run_command(...) Signed-off-by: Michal Simek <michal.simek@xilinx.com> Reviewed-by: Stephen Warren <swarren@nvidia.com>
* test/py: use space to interrupt autobootStephen Warren2016-02-26-1/+1
| | | | | | | | | | | | | | | Sending CTRL-C to QEMU's stdin aborts the process, even if stdin is being used as a serial port (at least in the raspi2 machine with "qemu -serial stdin"). Avoid sending CTRL-C to U-Boot to prevent it exiting. I'd originally used CTRL-C to make sure that if the character used to abort autoboot ended up being treated as part of a command as well, it'd abort command entry and return the prompt to a known state. However, this is not needed, since aborting the autoboot eats the character used to do that. Signed-off-by: Stephen Warren <swarren@wwwdotorg.org> Acked-by: Simon Glass <sjg@chromium.org>
* test/py: Add option to skip SPL signature checkingMichal Simek2016-02-25-1/+3
| | | | | | | | | | | | | | Provide user option to skip SPL signature verification for cases where u-boot is build with SPL support but full U-Boot is also verified without SPL. If you want to support this feature please add env__spl_skipped = True to your boardenv configuration file. For example Xilinx Zynq is using this feature where the same U-Boot binary is checked with SPL and without SPL(with FSBL). Signed-off-by: Michal Simek <michal.simek@xilinx.com>
* test/py: only check for SPL signature if SPL uses serial outputHeiko Schocher2016-02-24-2/+7
| | | | | | | | | check for U-Boot SPL signature only if SPL really has a serial output. So check if CONFIG_SPL_SERIAL_SUPPORT is active in board config. Signed-off-by: Heiko Schocher <hs@denx.de> Tested-by: Stephen Warren <swarren@nvidia.com> Reviewed-by: Stephen Warren <swarren@nvidia.com>
* test/py: put "Starting U-Boot" into separate log sectionStephen Warren2016-02-15-1/+3
| | | | | | | | | | | | | | | | | The initial boot of U-Boot happens within the context of the first test that needs to access the U-Boot console when there is no existing connection. This keeps all activity nestled within test execution, which fits well into the pytest model. However, this mingles the U-Boot startup logs with the execution of some test(s), which hides find the boundary between the two. To solve this, wrap the "Starting U-Boot" logic into a separate log section. If the user wishes, they can simply collapse this log section when viewing the HTML log, to concentrate purely on the test's own interaction. Signed-off-by: Stephen Warren <swarren@nvidia.com> Acked-by: Simon Glass <sjg@chromium.org>
* test/py: fix CONFIG_SPL testStephen Warren2016-02-15-1/+1
| | | | | | | | | | The Python ini file parser that's used to parse .config converts all keys to lower-case. Hence, all queries against the results must use lower-case. Fix u_boot_console.ensure_spawned() to test CONFIG_SPL correctly, or the connection will fail for boards that have SPL. Signed-off-by: Stephen Warren <swarren@nvidia.com> Acked-by: Simon Glass <sjg@chromium.org>
* test/py: capture the entire U-Boot version at bootStephen Warren2016-02-09-8/+3
| | | | | | | | | The existing regex simply ensures that the captured version string doesn't go past the end of a line. We really want to grab as much as possible. Do this by explicitly including a ) character at the end of the regex to match the last character of the version test. Signed-off-by: Stephen Warren <swarren@nvidia.com>
* test/py: support running sandbox under gdbserverStephen Warren2016-02-08-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implement command--line option --gdbserver COMM, which does two things: a) Run the sandbox process under gdbserver, using COMM as gdbserver's communication channel. b) Disables all timeouts, so that if U-Boot is halted under the debugger, tests don't fail. If the user gives up in the middle of a debugging session, they can simply CTRL-C the test script to abort it. This allows easy debugging of test failures without having to manually re-create the failure conditions. Usage is: Window 1: ./test/py/test.py --bd sandbox --gdbserver localhost:1234 Window 2: gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234' When using this option, it likely makes sense to use pytest's -k option to limit the set of tests that are executed. Simply running U-Boot directly under gdb (rather than gdbserver) was also considered. However, this was rejected because: a) gdb's output would then be processed by the test script, and likely confuse it causing false failures. b) pytest by default hides stdout from tests, which would prevent the user from interacting with gdb. While gdb can be told to redirect the debugee's stdio to a separate PTY, this would appear to leave gdb's stdio directed at the test scripts and the debugee's stdio directed elsewhere, which is the opposite of the desired effect. Perhaps some complicated PTY muxing and process hierarchy could invert this. However, the current scheme is simple to implement and use, so it doesn't seem worth complicating matters. c) Using gdbserver allows arbitrary debuggers to be used, even those with a GUI. If the test scripts invoked the debugger themselves, they'd have to know how to execute arbitary applications. While the user could hide this all in a wrapper script, this feels like extra complication. An interesting future idea might be a --gdb-screen option, which could spawn both U-Boot and gdb separately, and spawn the screen into a newly created window under screen. Similar options could be envisaged for creating a new xterm/... too. --gdbserver currently only supports sandbox, and not real hardware. That's primarily because the test hooks are responsible for all aspects of hardware control, so there's nothing for the test scripts themselves can do to enable gdbserver on real hardware. We might consider introducing a separate --disable-timeouts option to support use of debuggers on real hardware, and having --gdbserver imply that option. Signed-off-by: Stephen Warren <swarren@nvidia.com>
* test/py: detect another "bad pattern" in console outputStephen Warren2016-01-28-0/+2
| | | | | | | | | | | | Many error situations in U-Boot print the message: ### ERROR ### Please RESET the board ### Add this to the list of bad patterns the test system detects. One practical advantage of this change is to detect the case where sandbox is told to use a particular DTB file, and the file cannot be opened. Signed-off-by: Stephen Warren <swarren@nvidia.com> Acked-by: Simon Glass <sjg@chromium.org>
* test/py: check for bad patterns everywhere we waitStephen Warren2016-01-28-7/+19
| | | | | | | | | | | | | | | | Currently, bad patterns are only honored when executing a shell command. Other cases, such as the initial boot-up of U-Boot or when interacting with command output rather than gathering all output prior to the shell prompt, do not currently look for bad patterns in console output. This patch makes sure that bad patterns are honored everywhere. One benefit of this change is that if U-Boot sandbox fails to start up, the error message it emits can be caught immediately, rather than relying on a (long) timeout when waiting for the expected signon message and/or command prompt. Signed-off-by: Stephen Warren <swarren@nvidia.com> Acked-by: Simon Glass <sjg@chromium.org>
* test.py: calculate bad patterns on change onlyStephen Warren2016-01-28-24/+25
| | | | | | | | | | A future patch will use the bad_patterns array in multiple places. Rather than duplicating the code to calculate it, or even sharing it in a function and simply calling it redundantly when nothing has changed, only re-calculate the list when some change is made to it. This reduces work. Signed-off-by: Stephen Warren <swarren@nvidia.com> Acked-by: Simon Glass <sjg@chromium.org>
* test/py: use " for docstringsStephen Warren2016-01-28-24/+24
| | | | | | | | | | | Python's coding style docs indicate to use " not ' for docstrings. test/py has other violations of the coding style docs, since the docs specify a stranger style than I would expect, but nobody has complained about those yet:-) Signed-off-by: Stephen Warren <swarren@nvidia.com> Reviewed-by: Simon Glass <sjg@chromium.org>
* test/py: make crash detection more robustStephen Warren2016-01-28-13/+7
| | | | | | | | | | | | | | | | | | | | test/py contains logic to detect the target crashing and rebooting by searching the console output for a U-Boot signon message, which will presumably be emitted when the system boots after the crash/reset. Currently, this logic only searches for the exact signon message that was printed by the U-Boot version under test, upon the assumption that binary is written into flash, and hence will be the version booted after any reset. However, this is not a valid assumption; some test setups download the U-Boot-under-test into RAM and boot it from there, and in such a scenario an arbitrary U-Boot version may be located in flash and hence run after any reset. Fix the reset detection logic to match any U-Boot signon message. This prevents false negatives. Signed-off-by: Stephen Warren <swarren@nvidia.com> Acked-by: Simon Glass <sjg@chromium.org>
* test/py: move find_ram_base() into u_boot_utilsStephen Warren2016-01-28-37/+0
| | | | | | | | find_ram_base() is a shared utility function, not a core part of the U-Boot console interaction. Signed-off-by: Stephen Warren <swarren@nvidia.com> Acked-by: Simon Glass <sjg@chromium.org>
* test/py: add various utility codeStephen Warren2016-01-28-0/+19
| | | | | | | | Add various common utility functions. These will be used by a forthcoming re-written UMS test, and a brand-new DFU test. Signed-off-by: Stephen Warren <swarren@nvidia.com> Acked-by: Simon Glass <sjg@chromium.org>
* test/py: log when tests send CTRL-CStephen Warren2016-01-28-0/+1
| | | | | | | | | | Write a note to the log file when a test sends CTRL-C to U-Boot. This makes it easier to follow what's happening in the logs, especially since U-Boot doesn't echo the character back to its output, so there's no other signal of what's going on. Signed-off-by: Stephen Warren <swarren@nvidia.com> Acked-by: Simon Glass <sjg@chromium.org>
* test/py: drain console log at the end of any failed testStephen Warren2016-01-28-0/+38
| | | | | | | | | | | | | | | | Tests may fail for a number of reasons, and in particular for reasons other than a timeout waiting for U-Boot to print expected data. If the last operation that a failed test performs is not waiting for U-Boot to print something, then any trailing output from U-Boot during that test's operation will not be logged as part of that test, but rather either along with the next test, or even thrown away, potentiall hiding clues re: the test failure reason. Solve this by explicitly draining (and hence logging) the U-Boot output in the case of failed tests. Signed-off-by: Stephen Warren <swarren@nvidia.com> Acked-by: Simon Glass <sjg@chromium.org>
* test/py: move U-Boot respawn trigger to the test coreStephen Warren2016-01-28-2/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Prior to this change, U-Boot was lazilly (re-)spawned if/when a test attempted to interact with it, and no active connection existed. This approach was simple, yet had the disadvantage that U-Boot might be spawned in the middle of a test function, e.g. after the test had already performed actions such as creating data files, etc. In that case, this could cause the log to contain the sequence (1) some test logs, (2) U-Boot's boot process, (3) the rest of that test's logs. This isn't optimally readable. This issue will affect the upcoming DFU and enhanced UMS tests. This change converts u_boot_console to be a function-scoped fixture, so that pytest attempts to re-create the object for each test invocation. This allows the fixture factory function to ensure that U-Boot is spawned prior to every test. In practice, the same object is returned each time so there is essentially no additional overhead due to this change. This allows us to remove: - The explicit ensure_spawned() call from test_sleep, since the core now ensures that the spawn happens before the test code is executed. - The laxy calls to ensure_spawned() in the u_boot_console_* implementations. The one downside is that test_env's "state_ttest_env" fixture must be converted to a function-scoped fixture too, since a module-scoped fixture cannot use a function-scoped fixture. To avoid overhead, we use the same trick of returning the same object each time. Signed-off-by: Stephen Warren <swarren@nvidia.com> Acked-by: Simon Glass <sjg@chromium.org>
* test/py: Implement pytest infrastructureStephen Warren2016-01-20-0/+360
This tool aims to test U-Boot by executing U-Boot shell commands using the console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are: - Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect. - There is no need to write or embed test-related code into U-Boot itself. It is asserted that writing test-related code in Python is simpler and more flexible that writing it all in C. - It is reasonably simple to interact with U-Boot in this way. A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too. The hook scripts, relay control utilities, and udev rules I use for my own HW setup are published at https://github.com/swarren/uboot-test-hooks. See README.md for more details! Signed-off-by: Stephen Warren <swarren@wwwdotorg.org> Signed-off-by: Stephen Warren <swarren@nvidia.com> Tested-by: Michal Simek <michal.simek@xilinx.com> Tested-by: Simon Glass <sjg@chromium.org> Acked-by: Simon Glass <sjg@chromium.org> #v3