Mocking out AWS service in pytest references non-existent account - pytest

Question

I'm trying to set up a mock service call to kinesis firehose. I'm importing mock_firehose from moto and referencing it as #mock_firehose. In the test method I've created a client using boto3.
#mock_firehose
def test_push_to_stream(push_record, stream):
ret = app.push_to_stream(push_record, stream)
client = boto3.client('firehose', region_name='us-west-2)
I've exported the AWS_PROFILE I want to use and checked the credentials are correct. The error I encounter is:
botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the PutRecord operation: Firehose sample-name under account 123456789012 not found.
Apparently the dummy account 123456789012 is the default test account for running tests against mocked AWS services. I'm not sure if I need to create a stream for the test account, but that would make sense. It seems to fail if I comment out the boto3.client line and just having #mock_firehose above the method. Is there a setup step I'm missing requiring me to initialize a stream before calling #mock_firehose?

Answer 1

Moto is used to intercept any calls to AWS. Moto decodes the requests, figures out what you're trying to do and keeps an in-memory copy of the required infrastructure. So using Moto should make it seem like you're talking to AWS, but without the cost associated with it.
The dummy 123456789012 account is used to indicate that we're running against Moto, and to make sure it does not accidentally mutate any real infrastructure.
The ResourceNotFound-exception actually comes from Moto here. It knows that you have access to this test account, but it does know about the stream yet - because nothing has been created.
(AWS would probably respond with an AccessDenied error, saying that you do not have access to the dummy account number.)
With that in mind:
I've exported the AWS_PROFILE I want to use and checked the credentials are correct. The error I encounter is:
The credentials do not have to be correct, as AWS will never be reached.
To go one step further: credentials should not be correct, to ensure that AWS will never be reached.
I'm not sure if I need to create a stream for the test account
Yes, you should.
The general flow of a unit test using Moto looks like this:
Setup the equivalent infrastructure, that you expect to exist in AWS, in Moto
Run the business logic against Moto
Assert as required
So for your usecase, it would roughly look like this:
#mock_firehose
def test_push_to_stream(push_record, stream):
# Setup architecture that exists in Prod
client = boto3.client('firehose', region_name='us-west-2)
client.create_delivery_stream(...)
# Run business logic
ret = app.push_to_stream(push_record, stream)
# Verify that the records exist
...
As an implementation tip:
At the moment, Moto only sends Firehose records through if the endpoint is S3 or HTTP. Records to other endpoints, such as Elasticsearch and Redshift are not yet processed.
So it would make sense to setup your delivery stream to deliver to S3, as that makes it easy to verify whether the business logic send the correct records.
The documentation can be helpful to see more usage examples/ideas on how to use Moto correctly and safely: http://docs.getmoto.org/en/latest/docs/getting_started.html#moto-usage