What's happening in a for(auto& : container) loop?

[ElusiveTau]

I'm reading [1] which contains this snippet of code demonstrating the Specification design pattern:

struct BetterFilter : Filter<Product>
{
   // Filter method
   vector<Product*> filter(vector<Product*> items, Specification<Product> &spec) override
   {
      vector<Product*> result; 
      for (auto &p : items)
      {
         if (spec.is_satisfied(p))
            result.push_back(p);
      }
      return result;
   }
};

I get that auto is a type-inference type specifier and that it's referring to the objects inside the container `items` (a vector). Each vector item is of type `Product*` so `p` is a reference to a `Product*`.

But why declare a reference to a pointer and not just:

      for (auto p : items)
      {
         if (spec.is_satisfied(p))
            result.push_back(p);
      }

where the type of `p` is `Product*`?

Slightly off topic:

Can references be bound to a pointer? That's kind of weird because a pointer isn't an object -- I could be wrong here. As I understand it, you can use either a reference or a pointer to access an object so using a reference-to-a-pointer-to-a-Product to access the Product object seems a bit convoluted.

Sources
----------
[1] Dmitri Nestreuk's "Design Pattern"

[jonnin]

vectors have iterators, so in the first chunk, p will be an iterator (I could be wrong, but I think not; I get a little scrambled by auto still), which is very similar to a pointer in many ways but it is a little more than that. There are ways to get the type off of a variable, but I keep forgetting them; google it and you can see for yourself what things really are.

a pointer is not an object, but it exists, as does an 'int' which is also not an object in c++. Pointers are in fact unsigned word sized integers, or uint64_t type on most modern systems.

so yes, you can have a reference to a pointer, same as an integer, though neither are 'objects' in the OOP sense. At a higher level, looser sense, they could be considered to be objects.

[ElusiveTau]

There are ways to get the type off of a variable, but I keep forgetting them; google it and you can see for yourself what things really are

Ah yes. Good idea. typeid(p).name() should be helpful.
https://stackoverflow.com/a/20170989/3396951

Edited:

Meh. typeid(p).name() just gives a mangled name. It looks like auto& p and auto p have same type (a pointer to a Person).

// Ubuntu 20.04

#include <iostream>
#include <vector>

using namespace std;

class Person {
public:
   string name;
   Person(string name) : name(name) {}
};

static void GetTypeID(vector<Person*> people) {
   for (auto p : people) {
      cout << "autop: " << typeid(p).name() << endl; 
      cout << "    p = " << p->name << endl; // "autop: P6Person"
     // cout << "    p = " << p.name << endl; // Error: request for member 'name' in 'p', which is of pointer type 'Person*' (maybe you meant to use '->'?)
      break;
   }

   for (auto& p : people) {
      cout << "auto&p: " << typeid(p).name() << endl; // "auto&p: P6Person" 
      cout << "     p = " << p->name << endl;
      break; 
   } 
}

int main(int argc, char** argv) {
   vector<Person*> team {
      new Person("Alice"),
      new Person("Bianca"),
      new Person("Caitlyn")
   };

   GetTypeID(team);

   // Output
  /*
     autop: P6Person
       p = Alice
     auto&p: P6Person
       p = Alice
  */
   return 0;
}

[jonnin]

Ah, I remember (with some help from the web). (and if I had refreshed, mbozzi would have bailed me out below!)
p is the type inside the vector, of course, its your access point variable to USE inside the loop. It will be whatever <type> the container is made of.

the iteration is totally hidden, I keep forgetting that the variable and the iteration are not related.

[mbozzi]

The range-based for loop
for (auto elt : container) { /* ... */ }
has the English meaning "for each element elt in the container";
The type of elt is the value type of the container, Product*.

Range based loops are merely sugar for a more explicit loop:

// for (auto elt : container)
//   loop_statement();
// is defined as equivalent to the following code:
{
    auto && __range = container;
    auto __begin = __range.begin();
    auto __end = __range.end();
    for ( ; __begin != __end; ++__begin) {
        auto elt = *__begin;
        loop_statement();
    }
} 

Can references be bound to a pointer? That's kind of weird because a pointer isn't an object

The term object has a technical meaning in C++ that's substantially different from the object-oriented programming concept. In terms of C and C++'s object model, pointers (and ints) are indeed objects, and it's very possible to form references to them.

We can bind references to most things - if you must know, see this thread: http://www.cplusplus.com/forum/general/219156/

You should be able to use any debugger to inspect involved types.

[mbozzi]

It looks like auto& p and auto p have same type (a pointer to a Person).

Yes, except that auto& p binds the reference p to each element of the container while auto p incurs a copy of each element of the container.

(You can actually change the pointers stored in the container using the former.)

[ElusiveTau]

Playing around with it some more ... here's a difference:

#include <iostream>
#include <vector>
#include <typeinfo>

using namespace std;

class Person {
public:
   string name;
   Person(string name) : name(name) {}
};

static void RoboCopy(vector<Person> people)
{
    for (auto p : people) {
        p.name = "Robo_" + p.name; // Is this change visible to the caller? (No)
    }
    
    cout << "Inside RoboCopy:\n";
    for (auto p : people) {
        cout << "    " << p.name << endl; // Is the above change visible outside of the for-loop above? (No)
    }
}

static void RoboCopy_AutoRef(vector<Person> people)
{
    for (auto &p : people) {
        p.name = "Robo_" + p.name; // Is this change visible to the caller? (No)
    }
    
    cout << "Inside RoboCopy_AutoRef:\n";
    for (auto p : people) {
        cout << "    " << p.name << endl; // Is the above change visible outside of the for-loop above? (Yes)
    }
}

static void RoboCopy_PersonPtr(vector<Person*> people)
{
    for (auto p : people) {
        p->name = "Robo_" + p->name; // Is this change visible to the caller? (Yes)
    }
    
    cout << "Inside RoboCopy_PersonPtr:\n";
    for (auto p : people) {
        cout << "    " << p->name << endl; // Is the above change visible outside of the for-loop above? (Yes)
    }   
}

static void RoboCopy_AutoRef_PersonPtr(vector<Person*> people)
{
    for (auto &p : people) {
        p->name = "Robo_" + p->name; // Is this change visible to the caller? (Yes)
    }
    
    cout << "Inside RoboCopy_AutoRef_PersonPtr:\n";
    for (auto p : people) {
        cout << "    " << p->name << endl; // Is the above change visible outside of the for-loop above? (Yes)
    }
}

int main(int argc, char** argv) {
   vector<Person> team {
      Person("Alice"),
      Person("Bianca"),
      Person("Caitlyn")
   };
   
   RoboCopy(team);
   cout << "Outside RoboCopy:\n";
   for (auto p : team) {
      cout << "    " << p.name << endl;
   }
   
   RoboCopy_AutoRef(team);
   cout << "Outside RoboCopy_AutoRef:\n";
   for (auto p : team) {
      cout << "    " << p.name << endl;
   }

   vector<Person*> teamPtr {&team[0], &team[1], &team[2]};
   RoboCopy_PersonPtr(teamPtr);
   cout << "Outside RoboCopy_PersonPtr:\n";
   for (auto p : teamPtr) {
       cout << "    " << p->name << endl; 
   }
   
   RoboCopy_AutoRef_PersonPtr(teamPtr);
   cout << "Outside RoboCopy_AutoRef_PersonPtr:\n";
   for (auto p : teamPtr) {
       cout << "    " << p->name << endl; 
   }

   return 0;
}

Console Output

Inside RoboCopy:
    Alice
    Bianca
    Caitlyn
Outside RoboCopy:
    Alice
    Bianca
    Caitlyn
Inside RoboCopy_AutoRef:
    Robo_Alice
    Robo_Bianca
    Robo_Caitlyn
Outside RoboCopy_AutoRef:
    Alice
    Bianca
    Caitlyn
Inside RoboCopy_PersonPtr:
    Robo_Alice
    Robo_Bianca
    Robo_Caitlyn
Outside RoboCopy_PersonPtr:
    Robo_Alice
    Robo_Bianca
    Robo_Caitlyn
Inside RoboCopy_AutoRef_PersonPtr:
    Robo_Robo_Alice
    Robo_Robo_Bianca
    Robo_Robo_Caitlyn
Outside RoboCopy_AutoRef_PersonPtr:
    Robo_Robo_Alice
    Robo_Robo_Bianca
    Robo_Robo_Caitlyn

Remarks

1. There isn't a difference between for(auto& p:container) and for(auto p:container) if the container contains pointers (e.g., vector<Person*>). Changes to Person objects are visible to the caller.

2. There is a difference if the container just stores the object and not a pointer to those objects. for(auto& p:container) makes changes to Person objects visible in the immediate scope but for(auto p:container) does not. Changes to Person objects aren't visible to the caller if container is vector<Person>.

[TheIdeasMan]

I hope the following is not too confusing ....... :+)

The best and safest way to do a range based for loop is with an auto rvalue reference, as in:

for (auto&& item : container ) {
   // ... do something with item
}

This is good because it can handle lvalue references, rvalue references, const, and non const. This means it can achieve perfect forwarding.

Note, one can't have :

for (const auto&& item : container) // error

To enforce the const, send container as a const reference in a function argument.

http://thbecker.net/articles/rvalue_references/section_07.html

There is more to the differences in the syntax than scope and visibility, the time taken to make copies of objects is a factor too.

My take on how a reference may be implemented - or what a reference is: I think they are const pointers (not pointers to const objects), and const reference could be a const pointer to a const object, but they also have some TMP (template meta programming) associated with them. I have mentioned this before, and no one disagreed with me then - not sure if that makes it right though :+)

Some of my reasoning:

* References behave like pointers, it's more than just being another name for an object;
* A reference must refer to an object;
* In the <type_traits> header file there is TMP code such as
add_lvalue_reference
add_rvalue_reference

https://en.cppreference.com/w/cpp/types/add_reference

The code in <type_traits> header doesn't explicitly show how this built-in feature of the language is implemented, I am proposing that it might be done with const pointers.

[dhayden]

But why declare a reference to a pointer and not just:

      for (auto p : items)
      {
         if (spec.is_satisfied(p))
            result.push_back(p);
      }

You are right. The code should not use a reference. You should use a reference if you intend to change the value in the container. Use a const reference if the container's items are large so you can avoid making a copy of each item.

Also, it's a reference to the function parameter, which is a copy of the actual argument that was passed in the call. So even if you changed the pointer (not the value pointed to), it would just change the parameter, which gets destroyed when the function exits.

So whoever wrote the code used a reference where a copy is better, and copy where a reference is better. Sheesh.... The code would be better like this:

   // Filter method
   vector<Product*> filter(const vector<Product*> &items, Specification<Product> &spec) override
   {
      vector<Product*> result; 
      for (auto p : items)
      {
         if (spec.is_satisfied(p))
            result.push_back(p);
      }
      return result;
   }

1. There isn't a difference between for(auto& p:container) and for(auto p:container) if the container contains pointers (e.g., vector<Person*>). Changes to Person objects are visible to the caller.

Both let you change the Person pointed to, but auto &p lets you change the pointer in the vector itself, while auto p does not.

[mbozzi]

Note, one can't have :
for (const auto&& item : container) // error

Not typically (try it, e.g., where container is std::vector<bool>).

Sometimes, for (auto&& item : container) is the only viable choice when container is a highly generic type.

In typical application code, however, I think for (auto const& item: container) is usually a better choice because it strengthens the range-based loop's guarantee
"There's no funny business with loop indices!"
into
"There's no funny business with loop indices, and the container will remain unchanged!"

In many ways this is just another instance of a more general guideline, which is
"Prefer to create const references unless the referent is potentially modified."

This same guideline applies whether we use auto or not.

[TheIdeasMan]

mbozzi wrote:

Not typically (try it, e.g., where container is std::vector<bool>).

Hmm, every time I have tried with a std::vector of POD type, compilation failed. Is std::vector<bool> an exception to the norm? I never use vector<bool> always std::bitset instead.

mbozzi wrote:

In typical application code, however, I think for (auto const& item: container) is usually a better choice because it strengthens the range-based loop's guarantee
"There's no funny business with loop indices!"
into
"There's no funny business with loop indices, and the container will remain unchanged!"

In many ways this is just another instance of a more general guideline, which is
"Prefer to create const references unless the referent is potentially modified."

This same guideline applies whether we use auto or not.

What about this part:

TheIdeasMan wrote:

To enforce the const, send container as a const reference in a function argument.

*** note I should have said parameter, not argument.

as in:

void Function (const std::vector<Gadget>& MyData) {
   for (auto&& Item : MyData) { // Item is const lvalue ref to Gadget now
     // do something with Item 
   }
}

I guess if one forgets the const in the function parameter, then data could be changed - is that what you meant? :+)

I am guessing that it is not so much the funny business with loop indices, is it more that range based for loop will use an appropriate iterator? If a function parameter is marked const as above will that force the compiler to use a const iterator if one is defined, and error otherwise?

[mbozzi]

Hmm, every time I have tried with a std::vector of POD type, compilation failed. Is std::vector<bool> an exception to the norm? I never use vector<bool> always std::bitset instead.

Yes, it's an exception, because it uses proxy references. std::vector<bool> is a specialization that doesn't strictly meet the Standard's definition of "container" partly because its member type std::vector<bool>::reference is a class type instead of bool&.

As such, given a range-based for-loop like
for (auto const&& item: my_vector_of_bool)
the expression used to initialize the rvalue reference item in the de-sugared code is an rvalue:

{
    auto && __range = my_vector_of_bool;
    auto __begin = __range.begin();
    auto __end = __range.end();
    for ( ; __begin != __end; ++__begin) {
        // *__begin is an rvalue expression:
        auto const&& item = *__begin; 
        loop_statement();
    }
}

To enforce the const, send container as a const reference in a function argument.

That sounds like it would work fine, although of course sometimes container can't or shouldn't be const.

So if the choice is auto&& item the const-ness of item depends on the const-ness of the container, but if the choice is auto const& item item is always const. IMO the presence/absence of this particular const is a useful signifier whether the loop should modify (the elements of) its range.

[TheIdeasMan]

@mbozzi

Cheers, thanks for your always excellent advice :+)